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ABSTRACT 

The methodology for validating assessment techniques 
in a performance-based liberal arts curriculum was studied at Alyerno 
College. Methods for validating a generic competence instrument that 
measures "social interaction" were investigated. A generic instrument 
is, defined as. one that assesses a competence level across content 
areas instead of using a large varietyof instruments. The assessment 
system at the college requires students to demonstrate incremental 
chains y/hile progressing through sequential levels in each competency 
area. A comparison was under t^ken of the performance of 69 students 
on entrance to. college with that of 32 students who ha^ 2 years of 
college instruction on each of the social interaction dimensions 
(preparation, demonstration, self-assessment, arm leadership). 
Including students with a broad range of age and formal learning 
experience led to an effective strategy for ideriifying those social 
interaction behaviors that validate the constru rt. The analysis 
indicated that the study, of assessment techniques should not be 
limited to univariate methods. Attention is directed to instructional 
validity, construct Validity, instrument criteria validity, and a 
leadership dimension (the relat;ionship between i.eidership and type of 
program, age, and other college experience). (SV) • 
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ABSTMCT 



This report explore- i -u^^- i Lo the validation of more 

nontra'ditional assessmt 'jues, and tests some ways such 

studies may proceed. We e .^ -..j . ihe appropriateness of various 
methods for validating a generic competence instrument that 
measures Social Interaction, a construct with little or no history 
as a teachable college outcome or measure. We compare the 
performance of 69 uninstructed students on entrance to college with 
that of 32 sti^dents who had two years of college instruction on 
each of the Social Interaction dimensions (Preparation, 
Demonstration, Self-Assessment and' Leadership^ rind the specif:/ c 
dimension criteria. 

Results indicate similarities in- performance Ijetwe.bn tradi 'clonal 
age instructed students and mature uninstructed students. While' 
this may be expected i t . ^ilso indicates that ^roup comparisons may ■ 
not be an effective strategy for validating assessment t<^chniques 
if the ability is one developed through prior informal as well as 
college learning. Despite our efforts to do so, v;e were not able 
to control for tlie myriad range of variables uhat are l.kely to 
affect the results.. When performance of such an' ability also 
interacts witK a set of personal and ego , development variables, 
separating out the specific effects of instruction that show 
significant differences through group comparison is not an 
effective strategy, especially given the small sample, sizes 
generally available. 

However, some Social Interaction c'riteria did indeed separate the 
uninctructed students from the instructed, students wher. we 
combired all students in a discriminant analysis. These criteria 
are niore closely related to those aspects of Social Interaction 
that are learned as part of the more specific Social Interaction 
learning experiences. Thus, including students with a broad range 
of age and formal learning expe^'ience did lead to an effective 
strategy for identifying those Social Interaction behaviors tha.t 
validate the construct. Clearly, the s.tudy of assessment techniques 
should not be limited to univariate methods; patterns "of cohetjent 
group. performance provide us .with a more holistTc picture of 
performance, particularly of Social Interaction, not well understood 
and measured compared to some other 'abilities like communication^ . 

The present study outlines a procedure by which the^ integration of 
information about competence construct, different group character- 
istics and criteria evaluation contribute to an" information base for 
instructional development re-e^aluation of cQmp^ence definitions 
and revision of instrument criteria which measure these behaviors. 
The study helps to illuminate a key question in approaching the 
^validation of any faculty designed instrument measuring important 
but not well defined abilities new to higher-education instruction: - 
What strategies are appropriate, given where this instrument and 
construct are in their current development? 
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•VALIDATING ASSESSriEN'r TECHNIQUES IN AN OUTCOME-CENTEIiED. 
LIBERAL ARTS CURL CULlJM: SOCIAL u.TERACTION GENERIC " INSTRU^IENT 



Miriam Friedman Marcia Mentkowski Bernardin Deutsch 

M . Nicolette Shovar Zita Allen 

Office of Research St EvaiueiLion/.Socis^il Iritfiract.Lon Division 

ALVERNO COLLEGE 

i:^TRODUCTION 

r 

if * 

The pijrpose of this paper is to continue: to explore issues related 

C 

to the validation of Alverno*s more nontraditional .assessment techniques, 
and to empirically illustrate some ways in which such validation studies 
may proceed. Validation of assessment techniques is a corne T,tone in 
establishing thfe validity > abilities . d in col (ileiiLKowski 

& Doherty, 1977, 1983 ). This study of the Social Interaction competence 
is another example of our ^ork toward tl;iis goal. Other studies focused 
on the Communications and Valuing competences (Friedman, Mentkowski, Earley, 
Loacker and Diez, 1980) ^ the Integrated Competence Seminar (Assessment 
Committee/Of f. ce of Research and Evaluation , 1982 ), the Six Performance 
Characteristics Rating (Assessment Committee/Office of Research and Evalu^j 
tion, 1983 ), and a range of other instruments (Assessment Committee/ 
Office of Research and Evaluation, 1980). 

Several validation methods have emerged as ways to exainine assessment^ 
techniques in our previous work. In the present study, we investigate the 
extent to which the instrument measures the effects of instruction by 
comparing instructed . and uninstructed student performatice on each of 
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the Social Interaction dimensions as well as the specific criteria for 
each dimension. The second and perhaps more important objective is to 
examine patterns of student performance to assist in understanding 
student social interaction -behavior and the meaning of the Social Inter- 
action competence as it is currently defined. 

The present study examines these validation methods in relation to 
the Social Interaction Generic Criteria used to j udge performance at 
Social Interaction level 4 which students generally complete at the 
midpoint of their college career. In contrast to previous studies of 
Communications and Valuing where we studied levels 1 through 4, the 
Social Interaction Generic .Criteria assess level 4 only.^ 

Thus, the purpose of the current study is twofold. First, we 
study the appropriateness of various methods for validating generic 
competence instruments that assess constructs with liti ils 
) history as college out Aa' open-ended 

constructs are still uiiiier examination. Then we examine several 
study designs, such that constraints ^mposed by the educational 'setting 
, are recognized. Validation methods are considered fo^^ use at Alverno 
depending on how^they will work within the context of the curriculum 
and the setting in which the results are applied. We are examining ^ 
strategies that can be used within a regularly scheduled assessments 
process. Second, we are interested in adding to the understanding 
of how Socia' Interaction as an abii y develops during college, 
an low informal learning experiences prior to formal college 
learning coatribute to performance. 

Faculty observe differences in student performance over two years 
in college given the nature of the assessment process. That is, each 
(fpnsecutive .assessment is a "pretest** for the next competence level. 

Er|c ' ' 8 



Tbiis , im iutoresLing question is, How does the pertormance ot instructed 
students compare to unrnstructed students entering Alverno with more 
extensive lite experiences? 

First, experienced women can be expected to have social interaction 
abilities they have developed outside of college. Their social interaction 
skills m:iy have been developed informally, without the benefit of college, 
instruction. Thus, one interesting questioa has to do with how social 
interaction develops without the benefit of formal education. if we 
understand how it develops through experience, then we can better develop 
objectives and learning strategies that build on informal learning. To 
what extent should we expect that sorisal interaction abilities can be 
develops ■ especially ii rao i Lona4/-aged stucents? Do we 

expect raditiona aged students will develop abilities that matcli 

J 

t- ;e of the mature wom.va? Do we expect that abilities will be signi- 
ficantly more developed due to tl^e effects of instruction? If ego 
development or maturity is a major factor in the development of social 
interaction skills, t?o what extent should we expect that social inter- 
action skills can be deveLoped in college after tw^ years? 

Since we had been successful in demonstrating that traditional-aged, 
instructed students in the Weekday College Jiad performed significantly 
higher on Valuing and Communications level 4 criteria compared, to expec.i- 
enced, uninstructed students just entering Alverno in the^ Weekend College, 
we f^tlt we could expect that such differences wotild also be obtained 
for the Social Interaction competence. This was also expected because 
we consider Valuing to be an ability that is less likely to be "developed 
in the nontradi t iona 1-aged student through informal learning. Yet, at 
level 4, the nontraditional-aged students in 'Weekend College had not 
performed at level 4 as well as students with instruction. It should 



noted, however, that this wa^ true only ol' level 4, and not dI 
levels 1 to 3. Thus, nont radi t i ona 1 -aged students do develop some 
Valuing abilities through informal learning. In .our Coiiimuni cati ons 
^;tudy, we found that the uninstructed groHT>.jiemons t ra ted a different 
developmental sequence in Communications conW ten-ce , but with perfect 
sea 1 al) i 1 i ty . -We have reason, then, to [)elieve that nontrad i t i ona I -aged 
stu(it^nts do develop some of the , competences through informal learning. 

In the present study, we compare instructed students to unin- 
structr'd Jv.tudents. How iniV*bt we best d'es'ign sucii a study? TIh s (juestion 
IS particularly important because our first purpose is to examine 
validation methods that can be carried out within a particular curriculi'im 
design. For the Social Interaction competence, one o-f seve^ral constraints 
in carrying out a "true" p re- /pos ttes t design is that students at level 4 
are required to complete five Social Interaction assessments. 
V The Social Interaction Generic Criteria are used, to assess a stu- 
dent's de;.iOa^tration of her ability on five separate occasions in order 
for her to receive a level 4 validation (credential). In order to 
examine student per formance at level 4," one of these occasions must be 

chosen for a pre~/posttest design, one that is "content fair" (i.e., 
» 

students may. perform with or without a specific content background 
demanded by the instrument stimulus). 

One assessment occasion that is ^ntent-fair on which^uninstructed 
and instructed students may be" compared, is the Integrated Competence 
Seminar Group DiscussJ.on (Assessment Committee/Office of Research and ' 
Evaluation, 1982b). This instrument is content- fa i r , and would be 
likely to produce results generalizable across the college.^ (Note: In 



^With many measures, it is difficult to give a pre-post assessment with- 
out the student having some pre-training in vocabulary. Without that, we 
may be. testing just for vocabulary differences. 

^ " 10 



the Coimnunicat iop.s study, we controlled lor c out (Mil by solocting iiiiiii- 
structed students who had prior hackground in a subject area.) Since 
the instrument is content- t'a i r , we can administer it as a .pretest to un i - 
structed students. Currently, four of five Social Interaction instru- 
ments are content bound. The fiftli is an independent s^tudy ed as an 
external assessment. There are not enough students to provide us with a 
lai^ge enough sample size on this instrument* and it would also be diifi- 
cult to, give as a pre^-asses sment . 

Further, the ICS Group Discussion is currently the only social 
interaction performajice that is videotaped, and requires;, written 
as well as oral performance. In the present study, we felt it important 
not to elicit the Hawthorne effect, that is, that students v/ould be 

aware they were being observed as part of an instrument validation ^ 

-J 

study, as well as for credentia I ing . 

It was important to obtain a large enough sample size to yield 
statistically meaningful comparisons. In our first study. of the Valuing^ 
competence., we had been able to obtain performc-\ce data on only 11 
students who had contracted for validation at level A for comparison, 
with students completing the pretest. Choosing the ICS Group Discussion 
as the instrument for the study guaranteed at least 30 students in the 
instructed student group. 

In'sum, the ICS Group Discussion seemed the most appropriate instru- 
ment for the study, It is content^ fa i r , records of instructed student 
performance wei^e on videotape, we had accumulated enough records to give, 
.us. a large enough sample si^.e, and. records included written as ,wel 1 as 
videotaped oral performances. » . 
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in which 'the skills .nc drinoiis t laL (mIV SLudtMiLs (lemons I rat i socKii 
1 nltM-at-l 1 oris skills, alMn^ wittrolht-r skills ri a selling wht-rr tlit^y 
.irr not hrinv, t ro<lfMit. i ,i I ci , is perhaps hvsl Ic^l of Ihcir VihUily 

to translcM- their sooial i nt oi-aet i on al)ilit.y. Domoiist rat i ng tho i r 
ability in .t more lotiisod s.ftting shouM he more likely to elicit 
tluMT skills, t-^peeially ii Lheir pe r fo frnanee is boiu^ coniited towur'd 
a eompctenee va I i (Li t. i oe. or" OrtMtontial. 

UsiiiK the 1 n I e)/^ tsi U^'.i ('um}u-t. < ' ru f ' SiMnin.ii- (o-<>up 1) i seu!-,s i on mranl 
that instrncaed students' [)oriorrnance won hi he nieai:ured in the context 
of the remaining parts of the "ICS i nst rnira^nt , [n~Basket ami Oral Presen- 
tation, rhe instrncttMl student is exi)ected to denionsl rat e level A of 
Social Interaction aL Lhe same time that she is t;ei.ng assessed for Icv'el 
^ ol other compe LtMicc^s as^well. Thus, the instructed student would be 
challenged to^ perform a widt> range of abiLitif\s, not just Social Inter- 
action. It seemed inadvisable to ask entering .students to perform all 

three p.irts of the ICS, since bhe /rat iona 1 e , f o r tii<M r' coyip,! et i ng 

■ \ 1 - 

pret(^st was for'a Ic^ve] I validation in Social i ri te ra c t iojiVv^/1 iius , / t iiu^ , 
outcome is biased against the instructed students, rather than'' the . 
uniristructed student^s. Still another aspect of the ICS is it.Vuse m 
,the college for diagnostLc purposes rather than for c reden t ial ing . 
Only some students were , using the' ICS Group Discussion to complete ^ 
Level 4 validation. Thus, ndt only were we asking instructed ::>tudents 
to complete the Group Discussion in the context of ^he compieto Inte- 
grated Competence Seminar, we were also *asking that they transfer thevr ' 
social interaction abilities beyond the classroom setting to an external 
situa'iion where they were not being credentialed . 
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Since we had demonstrated that instructed students at level 4 
performed significantly higher in Communications and Valuing at level 4 
than uninstructed students, we. felt we could make the assumption that 
level 4 instructed students would perform higher on a level 4 Social 
Interaction assessment that was not asking them to integrate this ability 
with others. Instructors had already assessed these students. Each 
student had successfully consecutively mastered levels 1 to 3. Given 
this assumption, how would students perform on the ICS Group Discussion 
in a situation where they had to integrate and transfer their abilities, 
and where some students were not being credent ia led? We are also 
interested in how instructed students i^ompare with students of nontradi- 
tional age who had some informal learning in social interaction skills. 

Demonstrating that students can peVform social interaction skills 
after instruction is not the most important question, however, instructors 
report such increases in ability, and indeed, the fact tiiat students 
are validated on successive levels of Social Interaction indicates growth. 
We wished to make a comparison that would pit older women against 
younger women in a setting which would call for transfer of learning 
on the part of the younger, instructed women. On the face of it, 
this meant a test oi the differences in social interaction abilities 
between women who had experieaceMi instruction in the Weekday College, 
and un i iir.t rue ted women v^ho were just entering tlif* Weeken<l College. 

Fhe most important object iv<? is th(* eonstnict validity oi the 
sO( ial interaction emnpetenie. Tluis, it is mc)re i m[j(^ r t ;int that tlie 
group studied Ix^ a wide rai.iM* et slndeiits, with and without exjuMieiue. 
At the j;aine time, however, we .ire irneteslc<| in eXrHuiniiig the diltei'enees 
between inst nu ted antl nu i ris t r ue t (ul :;iit«ients, keeping in mind that the 
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Lininstructed groups will be given some prior instruction in vocabulary. 
Generally, our questions are: » 

• To what extent will instructed students differ in their social 
interaction performance? 

• Do they conceptualize social interaction differently as compared 
with uninstructed students? . ' ♦ 

• What patterns of student performance characterize these students 
These questions, more specifically stated in collaboration with the 

Social Interaction Division and the Office of Research and Evaluation, 
served as a guide for the study design: 

• How does instructed students' performance compare with uriin- 
strueted studerr. performance on each of the four Social Interaction 
dimensions of Preparation, Demoris L ra t i on , Se 1 f 'Assessment and Leadership 
( i nst ruct iona 1 va 1 idi ty ) ? . 

• How does instructed studerits' performance compare to unin- 
structed students on each of the behavioral criteria that comprise the 
dimens i ons ( ins t rumen t criteria va 1 idi ty ) ? 

~ To what extent 'do instrument criteria measure effects of 
ins truction? \ 

- Which criteria best discriminate instructed from uninstructed 
groups? 

- Are all important social interaetioii skills at \v.yi^\ k 
represented by the instrument's behavioral criteria? 

- Which criteria contribute most to (effective Preparation, 
Demonstration and Sel f-Assessment? 

• How might we better understand the meariing of the Social 
Interaction competence from an ariaLysis of students' i)erformanre irres- 
pective of difftM-ences lielwetMi instructed and tui i ns t ruct(Mi stU(i(Mits 
(construct va 1 idi ty )? 

- (^ui we id(»[itiiy patteriis of Social Interaction behaviors 
which charact(U-ize each ^rouj) of students? Does the Social 
inlerac-tion construct, as a dcMiiuMi set ol abilities, have 
the same meariing for dilferent groups of students? Are 
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different groups characterized by di f f erent s;r cial inter- 
action skills? ' ' / 
As we review patterns of performance, how/cai* 'e best 
describe the entry level of Social Interacti >n competence 
among uninstructed students who may have had. ii. formal 
learning experiences? 

How do patterns of performance in students wl.o are unin- 
structed compare to students who have been init^ucted? 
What relationships exist between Leadership a id the varLabl 
oi type of program, iru^truct ion , age and other lollcge expe 
ence? 
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METHOD . 

"/ 

Sam£le . *> 

Sixty-nine un i nslriicteti W(H^ke:\d Civllege women stucients, willi no 
prior Aiverno College experience, and 32 instructed Weekday College 
women students, with .it least two years of Aiverno College experience, 
comprise the sample. of students selected for l\\e study. Students ranged 
in age^ from 18 to T^A years with a niearipige of 32 years. Students varied 
on t[ie amo\rnt of prior college experience as well. They ranged from no 
prior college experience before entering Aiverno to [laving achieved 120 
credits of prior co 1 1 eg(^ expe r i ence be f o re attending Aiverno. 

'I'lie 69 Week(:n(i nn i ns t rue te(i students were drawn from all students 
e itering Aiverno Semester 11, January 1980 (n - 110). Seventeen of the 
110 students were re-cnitry stud(Hits arui were not included in the sample. 
Four students had previous Alvei'no (^o 1 1 egcexpe r i ence in th6 Weekday 
College; 10' did not participate tor various sclu^duling i:*easons»; 10 
completed the instrument but their performance was not usable for the 
con^)arison 'study becaust^ assessors did riot cc^mplete all the forms or 
because tlie recorder did not work. The remaining students (n = 69) 
comprised the final sample of un i ns t ructed, students. 

Tfie Wet^kday instructed group i nc ludt'd all 32 students who com- 
pleted the Integrated CompeteiHc Seminar in Spring, 1980, for whom 
complete videotaped pe r fo rmance^; wtM f avai lai)le. 

\ 

1 nstrimient 

The instrument used in tills study ot Social Interaction student per- 
(orinancc was drawn i roin tiuN Integralt^l (^ompet(uice Seminar' (ICS). One f)art 



u • 

of this Seminar^ Group Discussion, was used to assess student perfor- 
mance on competence levels 1, 2, 3 and 4 of Social Interaction. 

Group Discussion was given as -a pretest to uninstructed students in 
order to provide a pre-/posttest comparison for levels 1 to 4 oii the 
S'ocial Interaction f^eneric Criteria. Thus, uninstructed students experi- 
enced a special administration of the. same instrument administered to 

< 

the instructed students validated at level 4 of Social Interaction. 
While the Social Interacti^on Generic Criteria were not specifically 
designed for the ICS Group Discussion, this was the only naturally 

^1 

occurring opportunity to conduct a pre-/posttest study using a similar 

\. 

" — ^ ins't rument . Further, some instrucj:ed students attempted a level 4 

validation. That is, they were contracted for credentialing (contract 
group). Others completed . Group Discussion as part qf the ICS require- 
ment, but were not contracted for credentialing at Social Interaction 
level 4 (no contract group). 

The Integrated Competence Seminar (ICS)' is an assessment technique 
usually administered a u the mid-point of each Alver o student's college 
career. The purpose of the ICS is to provide an opportunity for the 
student to demonstrate how well she can integrate abilities she has 

/ ^ 

developed thus far, and to what extent she can t.ransfer thes^ abilities ( 

/' 
./ 

to a "realistic" situation. Thus, ICS is an out-of-class, cross-disci- 
plinary exerc i se ' assess ing a studtMit's ability to integrate level 4 of 
Communications, Problem Solving, Analysis, Social Interaction and Valuing. 

As a participant in the ICS assessment, the student takes on the 
r o 1 1» of a citizen d e c i s i o n -rna king I) o a r- d member, who niu s t ma k'e a s election 



Studies of thi5; natur^<* are designed to r(»s[)ect curriculum design. 
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The Integrated Competence Seminar (ICS) is an assessment technique 
usually administered at the mid-point of each Alve rno. student ' s college 
career. The purpose of the ICS is to provide an opportunity for the 
student to demonstrate how well she can integrate abilities she has 
developed thus far, and* to what extent she can transfer these -,ah i 1 i ti es 
to a "realistic" situation. Thus, ICS is an out-of-class , cross- 
discipUnary exercise assessing a s tudent ' s ab i 1 i ty to integrate level '4 
o£ Communications, Pftblem Solving, Analysis, Social Interaction and 
Valuing. 

As a participant in the ICS assessment, the student takes on the 
role of a citizeti decision-making board member, who must make a selection 
among three proposals for fuuding various city improvements. The student 
re[)resents one ol six areas needing improvement; health, cuiyire, 
recreation, echicatiou, vocational training, or environmental affairs. 

The student demonstrates her abilities in a four-hour exercise 
consisting of three parts: Oral Preset^tation , In-Basket and Group 
Discussion. For Oral Presentation, the student studies a set of back- 
ground inlormalion and delivers a speech to persu'ade the decigji on-maki iig 
board, to accept her choice among .three proposals. She then engages in 
an In-Baskel exercise which derives its name from the in-basket or tray 
on the manager's desk Ccontaining letters, memos, minutes and re|)orts 
awaiting action)., She solves prohjems, setfi priorities, analyzes, orgaiii 
and makes decisi(uu> as if she were encountering a real situation in her 
office. In Group Discussion, the student meets with other participants 
to discuss and make decisions on the final proposal, a common plan for 
the expenditure of (unds. ^ / 
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Procedure . ^ 

The present study examined student performance on one part of the 
Integrated Competence Seminar (ICS), Groim Discussion. This exercise is 
designed to rOi^asure interpersoftal andt/s'k-oriented skills in 
Social Interaction, Before the group discussion, students respond in 
writing to questions about their goals and expectations for: accomplish- 
ment in the exercise. Each group of five or six students then discusses 
t^t ee funding proposals and attempts to reach consensus on the one which 
would better serve the needs of the community. The group is initially 
leaderless. In the course of the half hour discussion, however, leaders 
emerge. After the exercise is completed, students are asked to 
reflect on their experience in relation to the other group members' 
performance . 

Following Group Discuss ion ,'^'^ssessors judge student performance 
against Social Interaction level 4 criteria. There were some differ- 
ences in the assessment procedure between the instructed and uninstruc- 
ted student groups in the present study. The instructed students' 
(Weekday' College) Group Discussions were videotaped and subsequently 
observed,^by two assessors who evaluated each student's performance and 
generated a summary statement describing the student's strengths and 
weaknesses as an effective group member. This procedure is staudardiz.ed , 
and used for assessing Social Interaction level 4 on the ICS. 

The uninstructed students' (Weekend College) Group Discussions were 
audiotaped. During each live group discussion, four to five assessors 
observed each group interaction. Each assessor observed one student 
primarily and one student secondarily. The assessors recorded stiadent 
behaviors during the ongoing discussion. After the discussion, the 
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likely to be unfamiliar with each other, with little or no prior 
•experience with one another in assessment situations. Students demon- 
strating their ability in class are likely^^to interact with students 
they know, and can more likely predict behavioral responses and patterns 
in a group situation. Finally, .in an assessment - situation in class, 
students are not asked to demonstrate Social Interaction skills in a 
context that includes all the activities demanded in the Integrated 
Competence Seminar. - ^ 

' While it is true that uninstructed students performing social 
interaction skills in the Group Discussion were being credentialed on 

r 

level 1 of Social Interaction, level 1 validation for incoming students 

is considered ,more of an exercise to identify strengths and weaknesses 

\ 

rather than an "assessment." Thus, uninstructed students' level of 
motivation is assumed to be similar to that of instructed students not 
contracted for credentialing at level 4. 




Social Inte raction Generic Criteria 

■% 

Performance on Social Interaction ^ levels 1, 2, 3 and 4 is coded on 

fqjjHf-^separate dimensions. The Prepa rat ion diniens ion "includes criteria- 

I 

for performance prior to Group Discussion: how she perceives her goals, 
the purpose of the task, etc. (See Table 1). The Preparation dimension 
criteria are directly related to the student's understanding of llie 
theoretical model- underlying behaviors exhibited during Group Discussion 
(8criteria). 

The Demonstration dimension encompasses all criteria relative to . 
the student's observable social interaction during Grou^) Discussion. 
Here the student is expected to demonstrate interpersonal as well as 
task-oriented skills (19 criteria). 

"Si 
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insure that uninstructed students would be abl\e to complate Group Dis- 
cussion. " . i 

The Integrated Competence Seminar offers' Social Interaction le^el 4 
validations for those students who have not completed all their valida- 
tions during their regular course work. Thus, some discussion groups 
were comprised of contractcj<d students who wejre interacting in a group 
where each person was beiiig credentialed. These students would seem 
to operate more out of self-interest and to be less group-oriented. 
Further, this contracted group may have been comprised of less able 
students, since they had not completed all Social Interaction valida- 
tions in class, and had to use the ICS Group Discussion opportunity for 
their last validation attempt. Consequently, the instructed contracted 
group may differ from the instructed group not contracted tolbe creden- 
tialed. There may be dissimilar motivational patterns operating which 
may injpact students* performance. The contracted group is assumed to 
have individual interest invested in their performance in addition to 
the motivation elicited by the group task. Since the contracted group 
was performing under the pressure of being credentialed, they may Jhave 
been less oriented toward group rather than individnal discussion out- 
comes . 

The ICS was designed to explore the transfer of abilities learned 

V 

in class to an out-of-class exercise resembling a more realistic situation 
It should be noted however, that transfer of learning from in-class 
Social Interaction demonstration may differ from the out-of-class ICS 
demonstration because students are asked not only to demonstrate Socia/1 
Interaction, hut also to integrate these skills with Problem Solving, 
Analysis and Valuing. Further, the ICS situation is associated with 
time pressure. Students who ;:)erform So(ial Interaction skills are 

2i 
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assessors evaluated students' performanc;e using evaluatioH^ forms tor 
level 1 and level 4 of Sbcial Interaction. The uninstructed students 
had just entered Alverno College, and were assessed by persons who*" 
we?re previously trained to evaluate student performance of entering 
students on level 1 criteria only. Therefore, these assessors experienced 
a brief training session prior to their participation on how to assess ; 
for level 4 criteria as well. The assessors'" evaluating uninstructed 
students' level 4 perf o rmance • we re probably not as exper^ienced as the 
level 4 assessors who evaluated inst ructed students ' performance. 

The Assessment Center administered the complete Integrated Competence 
Seminar to the instructed students. Uninstructea ^students were administ^re^d 
only Group Discussion. Assessment Center personnel- are experienced in 
administering the instruments. However, the administration to the 
uninstructed students was somewhat different from the instructed students 
in that the former group was larger than usual. It was also the first 
time that the Center administered the ICS with so many students at once', 
(n = 90). Usually the Center administers the instrument to 30 students 
at a time. ^ 

All incoming students usually attend a three-week preparation 
course (IN 050) before attempting competence validation of Social Inter- 
action level 1. In order to control the amount of instruction given tlie 
uninstructed group (who were being asked to perform prior to comt>^leting 
course work, during their second day in college), these students partici- 
pated in a brief introductory session prior to administration of Group 
Discussion. During this introductory session, the Social Interaction 
level 1 behavioral categories were introduced and explained (initiating 

2Z . 
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The Self-Assessment dimension include criteria for the student's 

/ . . 

] ability to reflect back on her own experience, to draw conclusions, and 
to recognize her social interaction strengths and weaknesses (5 criteria). 

The Leadership, dimension assesses the extent to which ^the student 
demonstrates leadership behaviors (0 criteria). 

The Social Interaction Generic Criteria are described on an evalua- 
tion form created to allow the assessor to score all students similarly 
for the purpose of the current stu^iy (See Table 1). 

P^rocedure for Coding Student ^Performance 

The coder was a faculty member in the Behavioral Sciences Division 
who wi^s also a member of the Social Interaction Competence Division. The 
assessor used th^ Social Interaction Generic Criteria Evaluation Form 
(Table 1). Because of differences in the dat:^l)ase between uninstructed 
and instructed students, the coder was aware of which^^students were 

# 

instructed and uninstructed. The'toder proceeded to score uninstructed 
group performance by analyzing audiotapes and evaluation forms submitted 
by assessors (reliability ^statistics, are unavailable). To score instruc- 
ted group performance, the coder analyzed the videotapes of ..^Group 
Discussion performance and assesso^* ayumm ry statements. Thus, the 
coder did not directly observe uninstrucyted group performance and 
if occasionally inferred level 4 behaviors from criteria ch.ecked by asses- 
sors on their "separate evaluation formjs. 

Using the Social Interaction Generic Criteria Evaluation Form, the 
coder checked all criteria the student demonstrated for the Preparation, 
Demonstration, and Self-Assessment dimens ions . \n addition, she rated 
each student on each dimensiori on a numerical scale ranging from "not at 
all effective" (1) to "very effective" (5). The coder did not base her 
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" Table 1 

Social iiiLcraQtion utmk Criterid Evdiuaiiaii form 



Date 



Name 



PREPARATION (?) 



Plans intercfction strategy, 
shows understanding of taslpll, 
^ shows preparation for content focus 
of the situation 

P I 2 3 4 '5 
Sot Bt all .Very 
effective. • ^fective 



P, 




_ perceivesTKe ta|jk goal as 
a' problem-solving 
situation 

identifies own position in 
relation to the task goal 

describes the key (Jimensions 
of the interaction setting 

develops a chronological 
■^strategy for assisting the 
"group toward |oal achieve- 
loent 



Pj) ^ clearly states the purpose 

of the task 



V 



relates theory to pr^tice 



\ 1 



PJ cites infonnation from 



outside sources 



Pg) * synthesizes the thinking 

of others 

Conmients; 
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DEriONSTMION (D) 



SELF-ASSESSMEtfl (S) 



HDC_ 
WEC" 



Demonstrates corajltioeht to goal achidvement, 
builds group cohesion, 

challenges and accepts challenges constructively 



D 1 2 
Not at all 
effective 



3 



4 5 
Very. 

effective 



V- 
V- 
V- 



V 



10' 

"ll' 

°12' 
\) 
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clearly states the purpose of the task 
reviews the task goal whenever appropriate 

sumnarlzes or evaluates group progress in relation 
to goal achievement • . 



D, ) stimulates 'discussion through effective goal-related. 



questioning 



shows enthusiasm for goal att^lnocnt through 
. non-verbal behavior (facial expression, 
body posture, voice tone) 

D^) ^ assigns^asks to help implement decisions or 
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Comments; 



attain consensus 

Mintains appropriate eye contact with everyone 
in the group 

uses both person-oriented and task-oriented skills 

uses a broad spectrum of Interaction skills 

chooses interaction skills appropriate to the situation 

attends to non-verbal messages and cues 

encourages those who exhibit withdrawing behavior 

challenges those who exhibit interfering behavior 

mediates differences by searching for common el&nts 

listens to others without interrupting when 
challengedt-with opposing points of view 

maintains an attitude of openness and responds flexibly 
when confronted with conflicting idk:as 

shows ability to modify ideas or opinions when approprlSt 

contributes a fair share of ideas and suggestions 

" '}' 
points out Implications of Ideas presented 



Demonstrates ability to accurately 
evaluate one's own social interaction 
effectiveness 

''SI 2:3 4 5 
Not at all Very 
effective effective 



3' — 



shows hov the demonstrated 
behavior had the intended effect 
on other (s) as 'specified In 
statement of goals -OR- 

provldes an adequate*, rationale for 
any changes made in the plan 

cites,, examples of effective 
behaviors and states how they 
affected other(s) in the group 

identifies strengths and weak- ' ^ 
nesses of own performance 

points out ways, to improve one's 
own effectiveness 



Comments: 



LEADERSHIP (L)^ 

L 1 2 3 4 5 
Not 'at all Very 
effective effective 

Comments: 



N ■ 19 

judgment on the -nuinber of criteria already checked for each dimensiQn, 

but rather, on her overall impression of the student's performance. The 

' , ( 

•Leadership dimension was scored' on a similar numerical scale, based on 

- /'' 
the coder's subjective judgment of which behaviors demonstrate leadership. 

A student was identi taed- as mastering a dimension if she was rated 

either a. 3, 4 or 5 'on the nutherical scale. ' 

r ' ■ 

P es ign ' ^ 

•The study design is to compare uninstructed students' performance 

to instructed students ' performance on Social Interaction Generic Criteria 

' ■ • " ' \ 
assessing -levels 1» 2, 3 and 4 of che Social Interaction competence, a 

two group comparison. 

Subsequent comparisons within each of the two groups were also 
planned. Following data collection, it became clear that uninstructed 
rtudents entering Weekend Tollege varied on age and prior college experi- 
ence. Consequently, students were categorized as "older" if they were 
age 27 to 54 years, and "younger" if they were age 19 to 25 years. The ^ 
uninstructed group differed in the amount of prior college experience, 
and were categorized as having prior college experience (range, 30 to 
120 credits) or no prior college (0 to 24 credits). Thus, three groups 
were identified: Group 1 (VPX , uninstructed, older, prior college); : 
Group n (Wr:C, uninstructed, older, no prior college); and Group 111 
(WEC, uninstructed, younger, no prioi) college). In order to control lor 
agfi?, the no prior college category included 9 students vlio had i^ome 
prior college (18 to 24 credits). Since the number of credits for these 
younger students was at the lower end of the distribution of credits, , 
the cutoff point for no prior college was set at 24 or less credits. 
The Weekday College instructed group was also examined for variation in 
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age (18 lo 49 y('ars) aiui prior college^ expericMuc (0 to 73 credits). 
Table 2 shows the number of sliKients in each group, llie mean, medjan and 
range lor age, and the ine.an, median and r.inge tor lujmher prior (re(iiLs 

Examination of Table 2 indicates that t.ronp I (WKC, niu nsl rucled , 
older, prior college) and Gronp 11 (WHO, un i iis t ructed , older, no prior 
college^ are comparable with respect to median age, but are dissimilar 
on median college experience. Conserpient 1 y , a comparison of Group 1 and 
TT can exanrine the extent to which prior college experience affects 
uninstrnraed students' performance on the Social Interaction Generic 
C r i te r i a . 

Further, Group 11 (WEC, uni nst ructed , older, no prior college) and 
Group 111 (WF.C, uninstructed , younger, no prior college) students are 
somewhat comparable in that Group 11 uninstructed students have no prior 
college experience, while Gronp 111 has an average of 8 credits with a 
median of 18 credits. Group 11 and 111 are dissimilar in age with 20 
years difference in the mean age and 10 years difference in med),an age. 
This suggests that a comparison between Group II and Group 111 could 
control somewhat for prior college experience, and allow us to 
examine the extent to which maturation (age) affects uniuiaructed 
students* performance on the Social interaction Generic Criteria. 

The Weekday College instructed group provides us an opportunity to 
examine the difference in performance between students contracted for 
credentialing at level 4 of Social Interaction (n = 9), and those who 
were not contracted (n^ 23). Examination of Table 2 shows- that median 
age and median prior college experience are almost identical for 
Group III (WEC, uninstructed, younger, no prior college) uninstructed 
students and for Group IV (WI)C , instructed, younger, no prior college, 
no contract) instructed students. This gronp comparison may provide 



Table 2 



Mean, Median and Range for Age and Prior college Experience (credits) for Unins true ted and 

Instructed Students Categori?:ed for Planned Comparisons 









Age 




Prior College Credits 


Grou]) 


n 


Ranee 


M 


Mdn 


Range _M Mdn 


Group I 

Weekend College (WEC) 

Uninstructed 

Older 

rrior college 


27 


27-44 


38.5 


33 


30-120 . 63.0 42 


Groiij) 1.1 

Weekend College (WEC) 

Uninstructed 

Older 

NO prior coiiegG 


25 




42.9 


33 


0 0 0 


Group III 

Weekend College (WEC) 

Uninstructed 

Younger 

No prior college 


18 


19-25 


22.5 


23 


0-24 8.2 18 


Group IV 

Weekday College (WCC) 

Instructed 

Younger 

No prior college 
No contractl 


23 


18-i9 


27.5 


24 

* 

i.' 


0-50 ' 20.9 18 

• 


'Group V 

Weekday College (WDC) 

Instructed 

Younger 

No prior, college 
Contract 


9 


19-4 (J 


30.2 


21 


0-73 . 26.8 23 



the best test of the effects of instruction on social interaction 
performance. Further, Groups II and III compared with Groups IV and V 
may provide a better comparison of the effects of instruction, since 
troup I has had extensive prior college experience. 

Following examination of the^effects of instruction, age and prior 

/ 

college experience, group performance will be examined to investigate 
the extent to which patterns in performance can be identified. These 
analyses will be used to interpert the" meaning' of the Social Interaction 
competence as a construct. While the groups may be somewhat different 
on several variables (age, prior college, instruction), students may 
behave similarly in response to the Social Interaction criteria. Different 
patterns in performance may suggest combinations of criteria that can 
illuminate the nature and meajiing of the competence. 

Analysis 

To examine the differences and interactions between the groups, 
t-tests and A.NOVA were employed. Discriminant analysis and Pearson 
product moment correlation matrices were used to investigate patterns of 
behaviors. Multiple regression and intra-item consistency tables examined 
the contribution of the behavioral criteria to the several dimensions of 
the Social Interaction competence. 



\ 



23 



23 



RESULTS 



The data analysis plan for the Social Interaction competence study 
was presented to the Social Interaction Division by the Office of Res(^arch 
Evaluation . The guidelines and questions for the data analys is were 



discussed and a set of priorities for the analysi s were generated , 

The Division decided to explore the following issues as a first 
pr i(^r ity ; 

I • How do instructed students perform compared to uninstructed 



studerjts on each of the four Social Interaction dimensions (instructional 
validity)? 



students on each of the behavioral criteria that comprise the dimensions 

(instrument criteria validity)? 

• How might we better understand the meaning of the Social 

Interaction competence from an analysis of students' Social Interaction 



performance, irrespective of differences between instructed and unin- 
structed students (construct validity)? 

Instructional Validity 

The purpose of the initial data analysis'is to examine the effects 
of instruction on student performance at Social Interaction level 4. We 
examined differences in performance between instructed and uninstructed 
student groups by comparing and contrasting Weekday and Weekend College 
students who differ in instructional experience, age, and prior college 



experience on each one of the Social Interaction dimensions (Preparation, 
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• How do instructed students perform compared to uninstructed 
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Demonstration, Seif-Assessm€?nt and Leadership). We then identified per- 
formance within Weekday College instructed groups which may be attributet^ 
to instruction. * 

Figures 1 through 5 describe the relative frequency distributions of 
dimension scores for each ol" the student groups compared. As menti^|^d 
earlier, a dimension score is comprised of a S-point scale which ranges 
from 1 (not at all effective) to 5 (very effective). A mastery cut-off 
score at point 3 of the scale was chosen hy the Social Interaction faculty 
scorer. Students who scored 3, A or 5 were classified by the scorer as 
mastery students and students who scored 1 or 2 were classified as no 
mastery students. 

Table 3 presents the percentages of mastery and no mastery students 
per group by dimension (Preparation, Demonstration, Self-Assessment and 
Le3<^ership) . Statistical significance tests of the difference in pro- 
portions (Glass and , Stanley, 1970). were employed to compare each group 
with each other group on mastery performance for three of the dimension 
scores^ (Leadership was not included).^ The only, signiiicant differences 
in proportions of students who mastered each dimension were found in 
the Preparation dimension. No significant differences were found in the 
Demonstration or Sc ment dimensions. In the Preparation dimen- 

sion, Gm?n T (VEC , uninstructed , older, prior college) had a signifi- 
cantly higher percentage of mastery students (P = 59.26) compared to 
Group II (WEC, uninstructed, older, no prior college) (P = 32.00) 



Since proportions were tested, results from mastery students will be ident 
to results from no mastery students^' thus, the latter were not compared. 
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Figure 1. Distributions of Integrated Competence Seminar scores for Croup I 
(WEC, unins'trncted, older, prior college): f^^astery and No Mastery 
students. ^ 
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Distributions of luLcKraLccl Competence Seminar scores lor iironp 
(WDC, instructed, younger, no prior colie^;e, no contract): Mastery and 
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Distributions of Integrated Competence Seminar scores for Group V 
(WDC, instructed, younger, no prior college, contract): Mastery 
and No Mastery students, 
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Prior college h prior collej^e No prior college h prior college No prior college 
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)1.85 

57.69 

'42.31 

68 ■,00 
32.00 



Preparation 
()8.00 55, 5f) 



32,00 



Demonstration 



1.00 



'} 

55. 5() 



Self-Assessment 



er :ip 



70.83 
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{z - 1.97, p' -05). The Weekend College groups are un i ns t ructed 
whereas the Weekday groups are instructed studerits. The Group III 
(VJECy unins tructed , younger, no prior college), and Group IV (WDC , 
instructed, younger, no prior college, no contract) comparison 
was examined more carefully, since Group III and IV seem to be the ' 
most comparable of all groups. The effectiveness of instruction can, 
of course, be more confidently explored when the groups compared 
are similar. Groups III (UTIC , un ins t ructed , younger, no prior 
college), and Group IV (WDC, instructed, younger, no prior college, no 
contract) (which allow for a better comparison of instructed and 
uninstructed students), did not differ significantly with respect to 
mastery in the Preparation dimension. 

Group V (WDC, instructed, younger, no prior college, contract) had 
a significantly higher percentage of mastery students (P = 11 .IS) com- 
pared to the other instructed students in Group IV (WDC, instructed, 
younger, no prior college, no contract) (P - 39-13) (z = 1.97, p'^.OS), 
and Group II (WEC, uninstructed, older, no prior college) (P = 32,00) 
(z - 2 . 37 , ^£ ,05) . Thus, Weekday students who contracted for creden- 
tialing (Group V: WDC, instructed, younger, no prior college, contract) 
demonstrated more effective Preparation compared to the Weekday students 
who were not contracted for c redent ia 1 ing^ (Group IV: WDC, instructed, 
younger, no prior college, no contract). 

Following comparison of students who mastered each of the dimensions, 
we examined the range of dimension scores (1-5) across :j 11 the groups. 
Table 4 shows the means and standard deviations per group by dimension. 
Paired t-tests were employed to investigate the mean differences between 
pairs of groups on each of the dimension scores. Table 5 shows the 
level of significance of t~test comparisons between groups. 
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■ Table 

Group Means and Standard Deviations for Each Social Interaction Dimension 





Group I 


Group II Group III 


Group IV 


Group V 




WEC' 


WEC WEC 


WDC 




Uninstnicted 


rninstructed Uiiinstructed 


Instructed 


Instructed 




Older 

Prior college 


i-'ier Youni;er 

}m orior college Ho prior college 


Younger 

No prior college 
No contract 


Younger 

No prior college 
Contract 


Students 


n = 27 


11 = 25 11 = 18 


n = 23 


11 = 9 






Preparation 






Mean 
Si) 


],% 


i 

Denonstration ; 


1.26 
I.IH 


l.B') 

O.fiO 


SI) 


3.11 

1 , Jl 


2,80 1.00 
1.12 I.J.! 

Sf'M 1 - .\ '.^(-^^ 1' 


1 10 
1.2'i 


I.J" 

0.97 


I'lenn 


IJ/l 


i.'li 1 ,^-)M.o,MlllilL 

!,o-i 


IM 


2.44 "\ 


SI) 


l,,:'h 


i.it 

! r'ulcrsliii^ 


1.12 


1.11 


Mean 


2.80 


2,f)) 2,8'i 


1,04 


1.22) 


Sj) 


1,29 


I.UH 


I 11 


1.2.8 
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Table 5 



Group Comparisons Showing Significant Differences from t- Tests for Each Dimension 





DIMENSION 




Preparation 


Demonstration 


Self-Assessment 


Group 


I 


II III IV V 


I 


II III IV V 


I 


II III IV V 


Group I 
WEC 














Uninstructed 
Older 

Prioij college 














Group n 
WEC 

Unlnstructeci 
Older 

No prior college 






.85 




.31 




Group III 
WEC 

Uninstructed 
Younger 

No prior college 


.80 


-.99 


.28 


-.49 


-1.07 


-1.29 


Group IV 
WDC 

Instructed 
Younger 

No prior college 
No contract ^ 


,91 


-1.01 ,05 


-.52 


4.33 -.M 


1 27 


t 


Group V 
WDC 


-1.15 


2.39* -1.78 -1.98* 


-.98 


-1.73 -1.14 -.48 


1.58 


1.27 2.20* .50 


Instructed 
Younger 

No prior college 
Contract 








] 
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As Table 5 indicates, significant differences between groups on 
mean dimension scores for Preparation were not different than the 
results obtained from tests of proportional differences. Group V 
(WDC, instructed, younger, no prior .college , contract), was signi- 
ficantly more effective in Preparation (M = 3.89) compared to Group II 
(WEC, uninstructed, older, no prior college) (M = 2.92), (t(Al) = 2.39, 

< .05) and Group IV (WDC, instructed, younger, no prior college, no 
contract) (M = 3.26; t(39) = -1.98, £ <.05). In the previous comparison 
of mastery students Weekend College, uninstructed, older women with 
extensive prior college experience (Group I) are similar in Preparation 
effectiveness to Group V (WDC, instructed, younger, no prior college, 
contract). The WEC uninstructed group (Group II: WEC, uninstructed, 
older, no prior college) (M = 2.92), without prior college experience,, 
is significantly less effective (t(41) = 2.02, £<.05) than the WEC 
uninstructed Group I (WEC. uninstructed,- older, prior college) (M = 3.56). 

Jhere were significant mean differences on the Self -Assessment 
dimension wnich did not appear in the tasks of proportional difference 
in mastery. Weekend young students (Group III: WEC, uninstructed, 
younger, no prior college) were significantly more effective: Self- 
Assessment of their performance (M ,= 3.59) compared to ai\y^.of the 
instructed groups (Group IV, M - 2,68) (Group V, M = 2.44) (t (39) = 
2.12, p <.05; t(34) = 2.20 ^ ".05, respectively). 

Since Group III (WEC, iHiiristructed , younger, no prior college) is 
somewhat similar to Group IV (WDC, instructed, younger, no prior college, 
no contract), we may conclude, with respect to Sel f -Assessment performance 
that instructional validity has not been demonstrated via most ot these 
comparisons. 
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One-way repeated measures ANOVAS were employed to investigate the 
main effect of group by dimension as well as the interaction effect 
between group and dimension. Such analyses assist our understanding of 
how" group differences in Social Interaction performance are affected by 
the variables ( ins tiucted/unins t ructed ; older/younger; contract/no 
cont ra ct ) . , 

Repeated measures AKfOVAS (2 levels of group by four levels of 
dimension)^ yielded no group main effects among all five paired comparisons. 
The group comparison which bfest examines the inst ructed/uninstructed 
comparison, Group III (WEC , uninstructed , younger, no prior college) and 
Group IV (WDC, instructed, younger, no prior college,. no contract), 
resulted in an interaction effect (F (17,22) = A. 75, £<.01) which > 
indicates that the direction of the differences between the two groups 
on each of the four dimensions is not consistent across all four dimen- 
sions. This interaction trffect (see Figure 6) may indicate the existence 
of other factors which affect student performance other than group 
membership and the dimension ,^being performed. When dimension means for 
Group III (WEC, uninstructed, younger, no prior college) and Group TV 
(WDC, instructed, younger, no prior college, no contract) are examined, 
it is clear that the direction of group performance means varies incon- 
sistently with type of dimension. Figure 6 shows that Group iV (WC, 
instructed, yoiniget', no prior college, no contract) performed signifi- 
cantly lower than Group 111 (WEC, uninstructed, younger, no prior 

^A two-group comparison was us<*d rather than a 5 x A design to investigate 
possible interaction rffetts i mmed i ;i te 1 y , hut also hccaus/:' the groups 
compared ditfer from each other v^'illi respect to t lie construct heiii^'. 
i nvest igated ( ins true t ed/un ins t rue t(-(l ; o 1 deM/younge r ; < ont rac t/no 
contract ) . 
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Group III (WLC, Unip.scructed, 
Youngtfr, Nu prior 
college) 

Croup IV Ci^^DC, Instructed, 
Younger, No prior 
college^ No contract) 




Prepara c ion Demonstration Self- Leadersliip 

AsseShmi-nt 

Social Interaction Dimension 

Figure 6. Interaction Effect Between Group III and CTroup IV 
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college) on Se I f -Assessment biit Group IV is higher (although nonsignifi- 
cant) than Group III on Demonstration and Leadership. 

In che other paired comparisons, a significant dimension effect (F 
(22,8) = 8.78, £ < .001) was obtained for Groups IV (WDC,- instructed, 
younger, no prior college, no contract) and V (WDC, instructed, younger, 
no prior college, contract) and Groups I (WEC , uninstructed , older, 
prior college) and IV (WDC, instructed, younger, no prior college, no 
contract) (F (26, 22) ^ 2, .11 , £<.05). 

In sum, there were no group main effects among the five comparisons. 
Students do not perform differently given their group membership. But 
there were two significant dimension main effects in two of the five 
comparisons. What differences in performance exist seem to be explained 
by the dimension the student is performing. Dimension affects student 
performance more than group membership. 

When dimension means were collapsed across Group IV (WDC, instructed, 
younger, no prior college, no contract) and Group V (WDC, instructed, 
younger, ao prior college, contract). Preparation (M - 3.45) was equal 
to Demonstration (M - 3.42). Both were significantly higher than Self- 
Assessment (M = 2.62). When dimension means were collapsed across Group I 
(WEC, uninstructed, older, prior college) and Group IV (WDC, instructed, . 
younger, no prior college, no contract). Preparation (M = 3.44) was not 
significantly higher than Demonstration (M = 3.25). Both were signifi- 
cantly higher than Self-Assessment (M = 2.94). 
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Instrmneixt Criteria Validity 

The second phase of the data analysis focused on the evaluation of 
the instrument criteria. 

• To what, extent do the instrument criteria measure the effects of 
ins t ruction? 

• Which criteria best discriminate instructed from unii||tructed 
groups? r * 

• Are all important Soe-kal Interaction skills at level 4 represented 
by the instrument's behavioral oriteria? 

• Which criteria contribute most to effective Preparation, Demonstra- 
tion, or Self-Assessment? 

In order to identify criteria that discriminate between instructed 
and uninstructed groups, we compared the frequency of response per 
criterion in one, group with that of*another via tests of association 
(chi-square) : Table 6 shows the percentage of students who responded to 
each criterion within each group. Eight two-group comparisons, and the 

significant chi-squares per criteria comparison are also indicated. We 

O 

found that few criteria were significantly associated within each 
two-group comparison. 

In the first comparison. Group III (WEG , unins t racted , younger, no 
prior college) vs. Group IV (WDC , instructed, younger, no prior college, 
no contract), the uninstructed group performed significantly higher on 
the fol lowing criteria: 

PA: Develops a cnronological strategy for 

assisting the group toward goal achievement 

D14: Mediates differences by searching for common elements 

SA: Identifies strengths and weaknesses of own performance 

S5: Points out ways to improve one*s own effectiveness 
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Table 6 

Criterion Response Frequendy Within Two-Group Comparisons 



1 



(continued on lUi page) 



Criterion 



94.4 94.4 00.0 50.0 83.3 11.1 83.3 11.1 50.0 21.8 44.4 33.3 lU 94.* ' 
91.3 13.9 00.0 8.1 13.9 2U 95.? 1?.4 «.5 2U 30.4 69.6 ?3.9 39.1 95,1 00.0 13.0 21.1 



111 
IV 

,2 ■ ■ 6.80** ' 



I ■ 

IV 

9 



88.9 88.9 00.0 48.1 II.8 25.9 88.9 40.] 6U 25.9 18.5 I7.8 81.5 29.6 7U iU 29 6 3U 
91.3 I3.9 00.0 8.M3.9 21.J 95.1 17.4 43.5 26.1 30.4 69.6 13.9 39.1 95.7 00.0 13.0 21,7 



7.42** 



II 

IV 



8U 88.0 00.0 40.0 68.0 8.0; 80.0 24.0 56.0 20.0 36.0 64.0 60.0 24 0 60.0' 12.0 20.0 3U 
91.3 73.9 00.0 8.7 73.9 21.7 95.7 17.4 43.5 26.1 30.4 69.6 73.9 39,1 95./ 00.0 li.U III 



4.70* 



6.72** 



IV 
V 



qn nno 8 7 73 9 21 7 95.7 17.4 43.5' 26.1 30.4 69.6 73.9 39.1 95.7 00.0 13.0 21.7 
'''^ "'^ oi 20:o 90.0 60.0 50.0 30.0 40.0 80.0 100.0 60.0 100.0 10.0 10.0 30.0 



90.0 90.0 10.0 10.( 



*£< .05 

**£ < .01 



4.14* 
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Table 6 Continued 



Group 

Comparison 


/ 






Criterion 










.^13 


\l 

. '14 15 16 


^7 ^18 ^19 




^2 


h h h 


III 

IV 


77. 8 16.7. 22,2 
91.3 13.0 21,7 


27.8* 55.6 77.8 
00.0 60.9V.56.5 


61.1:. 83.3 55.6 
52.2 37.0 '73.9 


38.9 
60.9 


44.4 
26.1 


72.2 83.3 38.9 
60.9 43.5 00.0 


2 

X 






.4.90* 

* 


i 

i 






5.16* 8.21** 


I 

IV 

2 


51.9 18.5 
91.3 13.0 

7.42** 


14.8 
21.7 


ll.l'55.6" 44.4" 51.9,85.2 51.9 
00.0 '60.9 •'56.5 52.2 87.0 73.9 

• 


51.9 
60.9 


33.3 

26U 


63.0 81.5 '33.3 
60.9 43.5' ()0.0 

. '6.22* 7.22** 


II 
IV 

) 

L. 

X 


56.0 24.0 
91.3 13.0 

5.88* 


20.0 
21.7 


16.0 48.0 48.0 
00.0 60.9 56.5 


44.0 .84.0 -52.0 
52.2 87.0 73.9 


4B.0 
60.9 


24.0 
26.1 


60.0 80.0 32.0 
60.9 4'15 00.0 

5 .35* 6.68*-^ 


IV 
V 


91.3 13.0 
80.0 00.0 


f.o 


00,0 60.9 56.5 
60.0 60.0 20.0 


52.2 87.0 73.9 
70.0 90.0 5G.0 


'60.9 
70.0 


26.1 
00.0 


60.9 43.5 00.0 
40.0 20.0 00.0 


2 

X 




1 


13.07 *** 











*£ < .05 ' 

''*p<.01 

o ^j**p ,',001 
ERIC • 



51 



0 



41 

The instructed group performed higher on the lOIInvv'lhg i jlirtiMH' 

D4: Stimulates discussion through effective goal related 
questions n 

In this comparison, no discriminative criteria were identified that 

could be attributed to instruction. 

In the second comparison, Group I (WEC , uninstructed , older, prior 

college) vs. Group IV^CWDC, instructed, younger.,^ no prior college, no 

contract), the uninstructed students performed significantly higher oa 

the following criteria: 

P4: Develops a chronological strategy for assisting the 
group towa rd go a 1 achievement 

S4: Identifies strengths and weaknesses of own performance 

/ 

S5: Points out ways to improve one^g-^'^own effectiveness 
The instructed students performed higher on the following criterion: 

Dll: Attends to nonverbal messages'and cues 
In the third comparison. Group II (WEC, uninstructed, older, no 
prior college) vs. Group IV (WDC , instructed, younger, no pric.- college, 
no contract), the uninstructed students performed higher on the following 
criteria: 

\ 

P4: Develops a i^b^-ono logical strategy for. assisting 
the group toward goal achievement 

S4: Identifies strengths and weaknesses of own performance 

S5 : Points out ways to improve one's own effectiveness 

Instructed students performed higher on these criteria: 

D7 : Maintains appropriate eye contact with everyone in 
the group 

Dll: Attends to non-verbal messages and cues 
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In the last comparison, between the two Weekday College Groups (IV 
and V), the contracted group performed significantly higher on these 
criteria: 

P8: Synthesizes the thinking of others 

D14: Mediates differences by searching for common elements 

All three unins.tructed WEC groups Groups I, II and III performed 

significantly higher on the following criteria: 

P4: Develops a chronological strategy for 

assisting the group toward goal achievement 

S4: Identifies strengths and weaknesses ^f own performance 

S5 : Points out ways to improve one^s own effectiveness 

The instructed, no contract group (Group IV) performed consistently higher 

on one criterion: 

Dll: Attends to non-verbal messages and cues 

If all Weekend College students are compiled into one . uninstructed 

group, and all Weekday College students into one instructed group (See 

i 

Table 7), the discriminative criteria that reflect the effects of instruc- 
tion are: 

P4: Develops a chronological strategy foj: assisting 
the group toward goal achievement 

P7: Cites information from outside sources 

D6 : Assigns tasks to help implement decisions or attain 
consensus 

07: Maintains appropriate eye contact with everyone 
in the group 

Dll: Attends to non-verbal messages and cues 

S4: Identifies strengths and weaknesses of own performance 

ft 

The dimension scores for eftective Preparation, Demonstration and 
Sel f -Assessment were entered in a multiple regression analysis to investi- 
gate which ci^i terion contributes the most to variation in' the dimension 
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seore. Kach criLerioii was corisidered as an independent varia[)]o and 
entered in a step -wise regreh;sLon. The dimension scores were used as 

;> 

the predictor variahies. Table 8 shows the multiple cor re I a t ion R , R*" 
an<i the ranks of the beta weiglits for eacTi entered criterion. 



Tnl) le 8 

' Multiple: Regression Summary Table for Behavioral Criteria 

Predicting; the Preoarat ion Dimension Score 

Multiple 2 ^i'l^ril ^ Simpl 
;t:ep Criterion R l< final step r 

i P C 1 e a r 1. y s t a t s t h l- p u r ii o s e 



of the task .39 .35 .AO .59 



P, Develops a chronological 

s t r a t gy f o r ass i. s t. i n g t lie 

group toward goal achievement .70 .49 .31 -^6 

P Perceives tlie task goal as a 

problem-solving situation .77 .60 .26 .51 

P Cites Infoniiation from 

^ outside sources .81 , .66 .24 .40 

P Relates theory to practice .84 .71 .24 .42 

6 



Since no sigrrificant differences were found among each criterion from one 
group compared to the same criterion in other groups, all students across 
the five groups were combined in the multiple regression analysis (n = 99). 



'['he f i V(» (riteria t.-ntertMi in the step-wiso rogrf'ssion expi<)in VO'/i 
ol" the variance in the i're{)a ra t ion diniension sii^rv. Criterion "eh'arJy 
states the purpose oi the task (P5)" contributes the most with ot 
the variance explained. Criterit)t) **i>erce.i ves the task goal, as a prohierii 
solving situation (PI),'- accounts for 11% of the variance. Criterion 
'Writes information from outside sources (P7)'* accounts for 6%, and 
c-riteriion "relates theory to practice (P6)" accounts for 5%, 

'Y\it' simple r" uidiCcites the correlation hetw^'en the criterion anrl 
the dimension score, and is a measure of the internal consistency ot the 
criterion. Thus, behavioral criteria which correlate highly wi tli the 
d i me n s i on s co re c 3 n also be cons i de re d to b e discriminative c r iter i a . 
Criteria "clearly states the purpose of the task (P5)" and "perceives 
the task goal as a problem solving situation (PI)" are thus good discrim- 
inative criteria (PS, r - .59; PI, r = .51). 

Table 9 

Mult I p 1 e Re g r e s s i o 1 1 S umma r y Table for Behavioral Criteria 
Predicting; the Demonstra t ion Dimension Score 



Mult iple ^ Beta in S iniplc 

Criterion R R final step r 



\), Stimulates discussion 

q 

through effective 

goal related questioning ,66 .43 .32 .66 

«• 

1)^. Siiows enthusiasm for goal 

attainment through 

nonverbal behavior .75 ,56 .30 .53 

^ Maintains an attitude of 

openness and responds 
flexibly when confronted 

with conflicting ideas ,80 .64 -26 .45 

Summarizes or evaluates 
progress in relation to 

goal achievement .84 .70 -26 ,43 



D^g Contributes 

of ideas and suggestions' .87 ,75 .26 ,58 



)utes c fair share , / 
das and s u g ge s t i o n s ' 

Ob 



The tivc criteria entered in Table 9 explain 75% of the vaiiance 
in the Demonstration dimension srore. Criterion '*s t iinu 1 a tes discussion 
through effective goal related questioning (D4j*' explains 43% of the 
Demonstration dimension variance and is also a good discriminative 
criterion (r ^ .66). Criterion contributes a fair share of ideas and 
suggestions (018)" and ''shows enthusiasm for goal attainment t [trough 
nonverbal behavior (D5)" are also good discriminative criteria, al.thougli 
i).'') e xp I a ins 1 1^% a n d D 1 8 o n 1 y 57o of t he v a r i a n c e . 

The five criteria entered in Table 10 explain 7 7% of the variance in 
the Sc 1 f -Assessment dimension score. Criterion *' cites examples of effec- 
tive behaviors and states how they affected others in the group (S3)'' and 
"identifies strengths and weaknesses of own performance (S4)" account for 
357c and 28% of the variance respectively and they are also good discrimi- 



native ite 



^ ms (S3, r = .60; S4, r ^ .5-9). 

Table 10 

Multiple Regression Summary Table for. Behavioral I Criteria 
Predicting; the' Se 1 f -Assessmen t Dlme[\sion Score 



Multiple ,^ ^t'* Simplt 
Criterion H"" final step _r 



S 2 C i Ll- s L-xamp 1 e. s o f e f f e c t i ve ...v 

b I h'l V io r s a a d s t a t e s h o w 

they affected other (s) 

in the group .60 .36 .46 .60 

* Identifies strengths and 
weaknesses of own 

p e r f 0 r ma nc e .80 .64 .41 .59 

Sj. Points out ways to improve 

one's own effectiveness .85 .72 .35 .47 



S ^ Shows how the demonstrated 

beha vio r had t he in tended 

effect on other(s) .86 .75 .33 .30 

S,^ Provides an adequate 

rationale for any changes 

made in the plan' .88 .77 .23 -.03 



Co a s t_r I ic_l^ V a 1 i <i i t y 

This section of the arial.ysi:; explores the coristMiri Vd ^ y of 



a n ability d e f 1 [ le d ? 

Questions rcpreseriti ng some of the construct validity issues arc- as 
f oi lows : 

• Can we identify patterns of soci.-.'l interaction behaviors which 
characterize each group of students? Does the Social Interaction 
construct as a defined set of abilities have the same meaning for 
diffc.^rerit groups? Are different groups characterized by different 
Social Interaction skills? 

• As we review patterns of performance , how can we describe the 
entry level of Social Interaction competence among uninstructed WEC 
students who may have had informal learning experiences? 

• How the patterns of performance in students who are 
uninstructed compare to students who have been ins'tructed? 

These questions allow us to look at the competence jmore holistically, 



rather than looking at specific dimension scores as we did in the previous 
univariate analysis. 

Multivariate analysis was used to examine these construct validity 
issues. A discriminant analysis (Klecka , 1975) was used to simultaneously 
examine all the Social Interaction criteria across the five groups. 
By exploring similarities and differences among Social Interaction 
behaviors across five different groups simultaneously, groups which vary 
with respect to instruction, age and other college experience, we can 
more bioadly describe the range of meaning of the Social Interaction 
competence. 



I he ' S o c i a 1 1 n t e r a c t i o n c o nip e t e i 



How best is Social Interaction as 
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■' Wo .ire aLlempting to identify the behaviors ' that best separate 
groups from one another. Once these behaviors are identified, we can 
use them to describe a group. if each group is described by a different 
set of be^iaviors, we nuiy infi>r that each group behaves differently 
because they conceptualize Social Interaction differently, 

F.ach behavioral crite>rion (0 or 1) including ea ch . d i mens i on score 
(1, 2, :3, 4 or 3) was included in the analysis. A total ot M) variables 
(witeri.i) wei-e mtered for r,\C\\ one oi ihc five groui)s, three Weekend 
College groups (1, 11, and 111) and two Weekday College groups (IV a[i(l V>. 
Four discriminant functions^ were generate>d. Table 11 summari/.es the 
statistics generated for oru* ol the four functions. 

Table 11 indicatt^s that functions f. and 2 account for 82% ol the 
variance and they both contribute significantly to the separation among 
the groups (funct.iof\ 1, p '^00^)); (function 2, p ".008). Thus only 

functions 1 ar\d 2 are usc>d to describe the groups. 



discriminant function is a set of numbers that can have either positive 
or negative values (coefficients). Each criterion is assigned a value which 
indicates the extent to which the criterion contributes to the similarities 
and differences between groups. The higher the value or coefficient; posi- 
tive or negative, the greater the contribution of the criterion to the 
separation of- the groups from each other. A positive or negative value 
indicates only the direction of the coefficient; it is not an rndex of 
the quality of performance. From a five group comparison, four functions 
can be generated. Those functions that are statistically significant 
are selected. 



5,7 
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C.'iiiOiiical Discriminant Functions for Five Groups Which Differ in 
■Program^, Instruction, Age and Other College Experience 





Percent 


Cdiulative 














of 


Percent of 


Canonical 


Wilks' 


Chi- 




Level of 


Function Eigenvalue 


Variance 


Variance 


Correlation 


Lambda 


squared 


df 


Significance 


im 


)3.93 


53.93 

^ 


^„ 

M 


^ .0380 


256.63 

^ V 1 V J 




0000 




28.38 


82.31 


,1% 


.16U 


U3.19 


105 


.0019 


3 jm 


12.82 


9).13 


,m 


.^368 


65.020 


68 


.5800 




^.8] 


100. 0(] 




M 


20.168 


33 


.5611 



'Vour [unctions can be generated frooi five groups. 
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T.ihlt' \2 iists the s I .ifi(l.i r j /(m! 'aru)iiitMl d i s( r i m i n.iwL Iimi(,Ii(.m 
coef f i ci euils tor each one of the behavioral criteria. 

TabJe 12 

Standardized Canonic, a 1 I) i scr imlnant 1-unct: ion 
CoefficienLs for the.l'irst: Two SiKnificant Functions 



Dimen;sion Criteria \ 
F F f f e c t i ve P r ej) a r a 1 1 on 

PI Perceives the task goal as a problem solving situation 
P2 IfkMitifies own position in relation to task goal 
F3 Describes the key dimensions of the interaction setting 
P4 Develops a chrono^Jogica 1 strategy for assisting the group 



P5 Clearly states the purpose of the task 

.ice 



outs ide sources 



P6 Relates theory to prac 

P7 Cites information from 

P8 Synthesizes the thinking of others 

D Ef^fect ivc Demonstration. 

Dl Clearly states the purpose of the task 

D2 Review the task goal whenever appropriate 

D3 Summarizes or evaluates group progress 

04 Stimulates discussion through effective questioning 

D5 Shows enthusiasm for goal attainment - nonverbal behavior 

D6 Assigns tasks to help,-,implemeiit decisions 

D7 Maintains appropriate eye contact 

D8 Uses both person-oriented and task-oriented skills 

DO Uses a broad S[)ectruin of interaction skills 



uruLion 1 


Function 2 


or { t i c- i ent s 


(-oe f { i (* i (Mil s 


-.837 


-.241 


.397 


.275 


.404 


-.431 


-.217 


.022 


.501 


.31 1 


-.006 


. 207 


- . 168 


. 325 


- .058 


-.024 


-.149 


- ; 688 


-1 .292 


- . 126 


.339 


-.291 


.267 


- . 191 


.203 


.465 


-.174 


-.648 


- .074 • 


-.467 


-.642 1 


-.150 


-.529 


-.130 


.230 

r 


.462 




-.627 
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I'a b i e 3 2 c o n I x. n u e d 






Dimension Criteria 


Function 1 
Coefficients 


Function 2 
Coefficients 


DIO 


Chooses interaction skills appropriate to the situation 


.240 


-.260 


Dll 


Attends to non-verbal messages 


.022 


.502 


D12 


Encourages those who exhibit withdrawing behavior 


.070 


.578 


D13 


Challenges those who exhibit interfering behavior 


. 141 


.252 


D14 


Mediates di f f erences 


-.528 


-. 182 


D15 


Listens to others without interrupting when challenged 


-.437 


■-.3i5 


D16 


Maintains an attitude of openness 


.799 


.462 


D17 


Shows ability to modify ideas when appropriate 


.11.5 


- .213 


D18 


Contributes a fair share of ideas 


.447 


-.213 


D19 


Points out implications of ideas presented 


. 160 


.674 


S 


Effective Self-Assessment 


-.389 


.820 


SI 


Shows how the demonstrated behavior had an effect 


-.092 


-.004 


S2 


Provides an adequate rationale for any changes made in 
plan 


.229 


-.235 


S3 


Cites examples of effective behaviors and states how they 
affected others 


.627 


- . 29 7 


S4 


Identifies strengths and weaknesses of own performance 


.740 


-.785 


S5 


Points out ways to improve one*s effectiveness 


.338 


-.572 


L 


Effective Leadership 


.395 


.677 
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For e.\ich function, the behavioral c-riteria wore rank ordtM^cd from high 



positive to h i ^Ji negative. Tlie criteria which contribute high posit ivt; 



function. The negative vjiues are^ more descriptive of the group which 
has a negative mean score on that function. ' 

The four highest positive values or coefficients and the four highest 
nogat; i ve C(m: f [ i c i enl s wei-e se ] (H:l.ed out f rom f unct i on 1 (Tab i c 1 2 ) . 



H i_gh p c) s i t iy e ^r iX e r i a o n f uri i o n 1 : 

D16 Maintaining attitude oi openness .799 

S4 Identifiers strengths and weaknesses o#^ own performance .7A0 

S'.] Cites examples of effective behaviors .627 

D9 Uses a broad spectrum of Interaction skills .552 

F4 Develops chronological strategy .501 

H i^h negative c r t e r i a on fu n c-t i on 1 : 

D Effective demonstration -1.292 

P Effective preparation -.837 

D6 Assigns tasks -.642 

D14 Mediates differences -.528 

D7 Maintains eye contact -.529 

High positive c r ijt e riji o n f lui c t j on 2 : 

S Effective self -assessment .820 

L Effective leadership .677 

D19 Points. out implications of ideas .674 

D12 Encourages those who exhibit withdrawing behavior .578 

Dll Attends to non-verbal messages .502 

H igh n ega ti v e criteria on func t ion 2 : 

54 Identifies strengths and weaknesses of own performance -.785 
P8 Synthesizes the thinking of others -.688 

D4 Stimulates discussion -.648 

D9 Uses a broad spectrum of interaction skills -.627 

55 Points out ways to improve own effectiveness -.572 



values descril)e the group having the higher positive !nean score on thai 



Those coefficients s(»lected "stand out 
twice or tlKree times as high. 



rom the rest, e.g., thejy riui 





• > J 

The nuN^uis of the two discriminant functions for the five groups wor(^ 
- 1 h e a c oni {) u t e ( i : \ 

F'unctiorx 1 Function 2 



Group 


I 


. A30 


-I^^IA 


(WEC, 


unins tructed , older, prior college) 






Group 


11 


1.31 6 


- .339 


(WEC, 


ins t r ucte(] , older, no prior college) 






Group 


III 


1 .354 


1.218 


(WEC, 


uninst ructed , younger, no- prior college) 






Group 


JV 


"K i62 


1 . t>2 I 


CWDC, 


instructed, younger, no prior college. 







no corit ra c t ) 

Group V -4.335 - .915 

(WDC , i n s I ni cl ed , y o unge r , no p r i, o r co 1 1 e ge , con I ra c t ) 

A plot of the means (centroids) is shown in Figure 7. This figure 
shov/s that the first discriminant function separates the three Weekend 
College Tlronps 1, II and III from the Weekday College Groups, IV and V. 
The contracted Group V is further from all the groups; Function 1 
accounts for 54% of the variance; thus we have more confidence in 
function 1 criteria as describing the groups. 

Figure 7 also shows the means with respect to function 2. 'The second 
'discr iijinant function separates the two groups who are younger (III and TV.) 
from Groups I, II and V. Function 2 accounts for 28% of the variance. 

A student who scored high positive op the first function is char ic~ 
terized by openness, ability, to reflect on her own experience in citing 
effective behafviors, using>.a broad spectrum of interaction skills, -and 
effectiveness in preparing by developing a chronological strategy. This 
profile of behaviors is more characteristic of , the Weekend College 
Groups I, II and III who scored high positive oh function 1. 

A student who scored high negative on the first function is charac- 
terized by e f f ec t i ve Demons t rat i on , e f f ect i ve Prepa ra t ion , as s i gn ing tasks , 
mediating differences, and maintaini'^g eye contact. These criteria are 
more characteristic of the Weekday College groups. the contracted group 
in particular scored high negative on function 1. 



er|c S3' 



6 



-6 



-8 



WDC 

Ir\st ructed 
Younger 

No prior colK-j^e 
No contract. 



Grouj.5 V 

1 us t r uc ted 
Youn>;t! r 
No prior c< 
Contract 



-6 



WEC 

Untnst ructed 
Younger 

No pr tcr col lege 



Group II 
WEC 

I'ninstructed 
Older 

No prior coIU 

G_rouj)_ I 
WFX 

Uainstruoted 
Older 

Prior college 



-4 -2 0 

Canonical D i scr i mi nan t function 



Figure 7. Group Centr'Uds With Respect to Function 1 and Function 2. 
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r't'Mjr to dale'! analysis, groups were c:a L t*>^<;) t' i zeii on Ifie basis of 
p r o g r a lu ( C / WT) C ') . 1' h f ( i i s r t" i iii i r i a n I analysis r ( • i 1 1 f < ; i ' c: e s 1 1 1 i s c a t. e go r i - 
7,citi<,)ii. VVi L[i I'cspf^ct tnii'.lion 1, , Wtv/kerui (.^olloge groups iiiglily 

reseniMe rach other ami W(H-kday groups higtily resemble each other 

Now let us eoneern ourselves v.'ith turn -lion 2. A person who scores 
higli i)osi.tive on fnuetion 'I is characterized by etfective Se j f -As ses sment , 
elieclive Leadership, poinliiig out i 114:1 1 i c/it i ons o 1. ifieas, ami attentive- 
no--... l.t^ gi oup uu-inbei s Uil tends lo nonvc-rbal iiu.'ssages , encour.ig(;s those 
wb.o exhibit withdrawing). These behaviors best describe Groups 111 (WEC, 
un i ns t ructed , younger, no prior college) and IV (V/DC, instructed, younger, 
no prior coiiege, no contract) wliich are the two younger student groujis. 
'Ihese yonngei' groups, from both Weekday and Veeken<.1 College, differ on 
i n s t r u c t i o n , h owe ve r . 

A person who scores high negative on function 2 will synthesize 
thinking of others, stimulate discussion, use a., broad spectrum of inter- 
action skills, identify her strengths and weaknesses and point out ways 
to improve her own e f f ect i vcMiess , This profile ol behaviors character- 
izes Group V (WI)C, instructed, younger, no prior college, contract), 
Group I (V,1lC, nninstructed , older, prior college), Group II (^//"EC, 
uninstructtMl, older, no prior college) and Group IV (WDC, instructed, 
younger, no prior college, no contract). ^ 

The classification analysis also, shows how students within groups 
are similar. Table 13 supports our categorization by depicting intra- 
group membership stability. The d i sc r iminant analysis confirmed that 
Weekday College students (Groups IV and V) are more similar among them- 
selves than they are to individuals in the other three? Weekend College 
groups (1, U, IJI). The highest pretUction for correct group membership 




Table 13 



\irccntage o\ Correct Classification lor F.ac.h Croup 



Actual (iroup 
Momberslilp 

Group I 
WEC 

Uaiiis true ted 
Oldt-^r 

Prior ci)lle;,',e 



Ml HI)' 



'red ic ted Croup Membership 



1. 1. 



[H 



0 o- 



IV 



1 



(!_rou_p^_ I I 
WEC 

I'n instructed 
Older 

No i)rior c^'l 



3 j;: " 



3 i:^ 



0 o: 



0 iU 



Croup 111 

wii 

Uiiiustrucied 
Youn^',er 

No pr ic'r (■>' i i e..',!.- 



<] 'I 



Crouj.) [\ 



Instructed 
Youiu;er 

No prior col Ir/.e 
No contract 

Croup V 
WDC 

lustruit'.'d 
Youiii'.er 

No pr ie>r I ! ' i.-i'r 
Coiit ract 
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classi t i calioii occurreul in Group [V (91% correctly classified) aruJ Group 
V (100% correctly c lass i L" i ed ) . 

riu* K^'oup categorization prior to our first analysis was made based 
oil diftereiices in age and other college experience for the uninstructed 
Weekend College students (Groups I, U, 111). The instructed Weekday 
College students were also divided according to cent ract/no-contract . 

In tiie previous luiivarLatt* analysis, age and prior college experience 
did not emerge as factors affecting Social Interaction performance. Tlie 
classification analysis shows, however, tliat intra-group similarities 
were liigher than inter-grcip similarities. This confirms our earlier 
( 1 <ist:i i i I ca t I ons . 

L(Md(.' rs ti ip Dimension Score Ana lysis 

Wi turthet investigate the meaning ot the So. Interaction cumpe- 

t^'iue, the {.■fNT<lerf;h i p dinienswui sc.orc^ was examined in r(^ 1 a t i cMish i p to 
t tie heliaviDral criteri/i. 

• What relationships exist hf^tween hend(»rship and type of program, 
i ns t fu: t i on , age and other college (experience? 
Co r t{' ] .1 1 ion matrices tor t\'ich grouj) w(^re g(MHMaled to idenliry 
''''t]a"M<i 11 •i)t«'ti.. wh 1 rli r(M"fel<ite h i 1 y With t tie Leaders[ii[) <l i mens i nri 

uiUiiri o.n h ^'lonp, .uul ifleutily wh i i h <>| tlirsf' h i g,h 1 y ^^'iirLited 

i -ii ;i V 1 o r :. hold i unst.Hi( .mm*!:;. >.M"'''Hps ( :.re T^h I r P'U . 

I hr h»di,jvi(iis win <li yi*')<h>d high i n t o r - ( o r r." I .i t. t on:; (p .01) will, 
l.o.i'h I *:h i p :;fo { f-:. t j t espn I t vo t» 1 oiip iiiPiithe r s h i p .\\ c : 



M 1 ir'ii I 1 ( 



I I MIS s I nil Mil ( >ii>di I • f I <" ( 1 I vo g.oa I i r 1 .i t r( I 



I p " : : t I (M 1 1 n 



t n I r r . I ( I 1 , 1 M : ; I: i M : , . i p o i o p i I a t i ' t ( - 1 1 1 1 • 



> 1 I 1 1 . 1 1 ) t » 1 1 



i H i'"'. 1 .f >n 1 I I h II t f ,1 } ,M i "Jm i r n i i (h-.i ;md s u ^',p,o s t i (Ui 
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Group V (WDC, instrucLcd, younger, no prior college, contract) had 
very few behaviors which correlate highly with Leadership scores (8 out 
of :yz) . The nature ol the exercise in which students are c redeiit i a I ed 
tor level A validation may have affected the frequency occurrence ot 
Leadership behaviors. The* otht?r groups had an average of 20 beliavioral 
criteria tliat are highly correlated with the Leadership dimension score. 
Table lA shows the list of criteric. which are highly correlated with tae 
Leadership dimension score. 

As mentioned earlier, Group V (WDC, instructed, younger, no prior 
colleg(», contract) students were expected to demonstrate mure st»lf- 
iuitiatiHl behaviors be(*ause tlu^y were contract(Mi. 

if Grnuj) V (WI)L\ i [IS t r-jc t(Mj , younger, no [)rior c..)[lege, contract) 
is discanied from the Leadershij) criteria correiaticMi matrix, otluM' 
f)(MMVj<^rs niay ;i t so b(^ atlded to list of brliavicn'al oritoria which 

highly correlatt* with Leadeiship. These Leadership critt^ria relatic^n- 
ships ho 1 1 1 a c ro s s g r o u p s . 

The add i t i ona 1 behav i ors a re : 

P6 Re Kites theory to practic:i? 

1)1 CI ea r 1 y s t a tes t he purpose o i t he t as k 

I)' Suimna r i /e or (*v.i 1 uat <»s group progress in relati(Ui lo 
>M><i 1 a (. h i e veiiien t 

1)() Assiy.',us tasks lo hel|i imp I omen t dot ision^i or <i I t <'m u 
consensus 

I)'^ l!:;os ■] hru.id ;;p<.M. t rum of inlor-wl ion ?;kilh; 

\^\2 i\fn-oui ago:; t lior;o wfrn oxhihit w i t hd r ,iw i n li; [)oh,ivior 
A Lo.niriidnp diiuon:;ioii »'inrr>N':; fioin t fi i f o r ro 1 a t i ( ui .i I iua lysis, 
S i m r sr)iri*' Lrailo j ' ! i i p i i i 1 «' r" i a hn 1 d ( ous t an I a r i t is all g r ( aip:. ( r X( 1 ud i ng 
Group V ) , wr may < on< 1 udo that t hoso b< hav i o rsi I c i i t o r i a do ( j i hi n ^ 
l/M do rs in I ^ are )m Mie r i o r a t ho r t i la r i situation or ou p s po t M i r , Tlio 



Table lA 



Behavioral Criteria Within Groups Which Were Significantly Correlated with the 

Leadership Dimension Score 



^^3 
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Group V 
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Uninstructed 


Uninstructed 


Instructed 
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Older 


Younger 


Younger 
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No prior college 
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■•7 


D2 ' 


D4 
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Di9 
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geru»ral description of a leader thus includes person- and Lask-or i ented 
skills. A student demonstrating person-oriented behaviors is sensitive 
to group members' neecis , and ases a wide range ot interaction skills 
appropriately. A student demonstrating task-oriented skills stimulates 
discussion, contributes ideas, cites information summarized and assigns 
tasks. She takes on an active role in the group discussion. 




CONCLUSIONS 



Inst ructional Vali dity 

Findings frum the group comparisons indicated that instructe^d and 
uninstructed students who completed th,^ Integrated Competence Se^mi.nar 
perform similarly on each one of the Social Interaction dimensions: 

I:' 

Preparation, Demonstration, and Self -Assessmerit . Some significant 
differences were found on the Preparation and Self -Assessment dimensions. 
The directions of the differences, however, were not consistent enough 
to support the iristriict iona L validity of the Social Interaction competence. 
No differences were obtained on t lie Demonstration dimension, wliich represents 
the performance aspect of tlie Social Interaction competence. 

Effective performance on Preparation is reflected in the degree lo 
which a student prepares to present her itieas (hiririp, the Group Discus- 
sion. Apparently, the motivation to achieve a Level 4 validation 
enhanced the performance of the contracted group on the Preparation 
di mens ion . 

Oru:' of the youriger uriinstructed groups was most effective in identi- 
fying strengths and weaknesses in their own performance and pointing out 
ways to imi)rove Social Interaction effectiveness. No re 1 a t i ons h i f)s were 
found ht? twee n effective Preparation and effective n(Miu)ris t ra t i on or lie t we en 
effective Se 1 1 -Assessment and effect ive Demonstration. It is assumrd, 
however, that learning activities which include groat er emphasis on 
Preparalio[i and Self-Assessment skills may assist students in their 
Demonstration pe i" f o rmarice , thus assisting students lo plan strategies 
[)rior to group interaction, and lo dii'ect their attention Vo /isso!;sing 
thoir perlonnance alter completing the task. 
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The ANOVA findings did not yield group main effects. These results 
support our previous finding of similarity of performance among groups, 
and no consistent direction of higher performance by the instructed 
group. The dimension main effect shows intra-group differences rather 
than inter-group differences on ttie various dimensions. 

There are some possible explanations for these findings. One 
centers on the validity of tiie Integrated Competence Seminar as an 
adequate measure of level 4 performance, because of the relationship 
of the Integrated Competence Seminar (ICS) assessment hnique to 
Social Interaction learning experiences. During the first two years 
of her etiucatlon, a student is expected to accomplish one validation 
at level 1, three vali(iations at level 2, one validation at level 3, and 
five validations at level 4 in order to complete her Social Interaction 
requirements. Tlie student is credentia le(i wilhin various classes which 
ofier Social Interaction validations. The ICS ass(*ssment procedure for 
most students is their first optni-ended, unc reden t i aled , unstructured 
experience. The student is expeqted to transfer her Social Interaction 
abilities to the ICS "realistic situation." Tins is her first alteiiipt 
to translate her ac:(juir(Mi abilities into an out -o f ~c lass experience and 
test the "outcomes" of her learning expe r if^ices . Thus far, she lias been 
invoIv(ul in the proc(.^ss of acquiring the desired i)ehaviors in clas^;. Now 
she is expruted to demonstrate her ability and integrate prev.ious learnings 
in <in unknown situation. She is asked to move from "guirled iiuUrue- 
tiori" to sf^ I f- i n i t i ated activity. In that respect, slie is more 
similar t ban d i f f e ren t i roin the iru^oming students who are a h;o 
encountering an rf|)e)i-(,'nded t a s k-o r i en t cmI grouj) <li^.cnssion lor tlie 
I I r s I [ 1 me . 
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The* study pointed out that at this juncture instructed students do 



ted students (Weekday) achieved their Social Interaction validation 
requirements in classes prior to tlieir ICS assessment, one may postulate 
that the problem of lack of instructional validity lies in the issue of 
trans fer . Tf faculty have evidence that social interaction skills 
learned in classes are effective and sufficient in enhancing Social 
Interaction performance within any given situation, the problem of 
adequate learning activities for successful trans fe r should be 
explored. Several approaches could be considered: 

• Implement a variety of simulatiqu exeicises of an open-ended 
na t ure throughout the four yea rs in col 1 ege . 

• Emphasize open-ended simulation exercises at the advanced 
levels of Social Interaction activities (e.g., mainly during 
the last two years). 

» Prepare the students for ICS-type activities by reducing 
structured learning in class from the beginning. 
Comparing instructed and un ins true ted students on a "real life" 
exercise after only two years in college may be a premature attempt to 
demonsLraLe instructional validity. The (jiiestion here may not be 
instructional validity but rat lie r validity of transfer, especially 
with the Social Interaction competence, where an ins true ted studf^nt^; 
encounter similar situations almost daily. If we assume that Alverno 
students are provided with the necessary tools to interact effectively, 
an additional step is needed to ensure that ti[ies(» tools can he brought 
in a variety o f o u t - o f - c 1 a s s situations . 



not differ significantly from nninstructed students. Since most instrnc- 
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Another observation has to do with the extent to which the Group 
Discussion in the Integrated Competence Seminar elicited higher level j 
cognitive performance and content integration. The fact that the 
Weekday College instructed students performed as well as more mature, 
uniustructed students in the Weekend College may show that Social 
Interaction can be taught. No d i f f ere nces between the £r()ii£s is 
a posi tive result. But it may also be the case that it is not until 
levels 5 and 6 that differences, between uniustructed and instructed 
students show up. Still, the Social Interaction Division may need 
to be more selective iu- ident i f y ing criteria at level 4, criteria that 
expect higher level cognitive performance and content integration. 
The iritegrat^ion of content and competence, which is not measured by 
the "conte/it- f a i r" Integrated Competence Seminar, and the ability to 
analyze behaviors and group behaviors may be the difft^rence l)etweer) 
uniustructed and instructed students. Given thf^se results, it may be 
that the Preparation dimension is the key to the differenctv 
Preparation and Self-Assessment skills may devtMop iirst. 

Construct Validity 

As stated earlier in the design section, dilfertuit Socia! f te ra c t i <hi 
performance [patterns may imi)ly that different constructs are crnoloyed l)y 
students from diff circuit groups in producing desired beiu^viors. Patterns 
of behaviors which fiold constant across dissimilar groups may ii<iic<jL(* a 
generic c^ons truest. Tlu^ (pu\stion of tM)nstrnct validity as it is r (Ma ted 
to diffr'reut groups, instructed vs. uu i us t r luH cd , as w.-ll is '.ilfereul 
l)a ckgr^ouiuis , is p.i r t i c n 1 a i" 1 y importruit Vv-hen ouv view^; constrct v.ilidi^.y 
.IS iocusing on respunso .muI i ii I i m p re t a t, i on oj storfv; (Mf;:;, irK I'WS), 
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whore the* validity oi interpretation is the main emphasis rather than 
the validity of the instrument. 

As Nitko (1980) states,: "Following the cogni tive- psycho logy views 
summarized by Chi and G laser (in press) one may find, for exam])ie, that 
experts ?nd masters exhibit different behaviors and may use differtMit 
internal cognitive ])rocesses when attempting to answer the same test 
items. Measuring these kinds of differences sharply contrasts with a 
measurement model that is linked to a conception of niastery ttiat is 
operational ized as a certain proportion of correct answers on a well- 
defined pool of items." ' 

A study design wliich incorporates five dissimilar groups described 
by a wide range of different variables may assist faculty in understand- 
ing the differences as well as similarities in Social Interaction 
l)ehaviors which characterize each group. Can wc identify patterns of 
Social Interaction behavior wliich characterize each group of students? 
Does the Social interaction construct as a defined set of abilities liave 
the same meaning for different groups, or rather, are different groups 
identified by different Social Interaction skills? 

Since the univariate analysis did not yield major differences among 
pairs of gi-oups, it is particularly irite res ting to set^ what kind of 
differences emerge in a multivariate analysis which takes the inter- 
corielations among the variables into account. I'lie iiH^ans of the I i vc^ 
gr'>n[),s with respect to thc^ first discriminant fujution, which was highly 
significant and cont i' i buted 'y'.^ i>fM"ceiit t(» tlic sepa rati oil .unong the groups, 
intlicatcMl (in spite ol the previous findings) that the three un i ns t tuc te<l 
groups exhibit(Mi closer i)i'oxinnty on the irui 1 t i va i' i a t e space while the iwc 
i [IS t rue t(Ml >^rc)Ups were also el(Kse in I iie opposite <1 i ret t. i O!) . That m.iy 
inii)ly similar underlying cc)ns t rn c t .s eniploye<l fjy t. iiree nn i lu; t i uc t, ed 



77 



66 

groups, and similar underlying constructs employed by the instructed 
groups. Since the? major difference between the two groups was the 
instructional factor, one may postulate that behaviors which discriminate 
the unanstructed groups are descriptive of baseline behaviors prior to 
instruction, whereas behavi()rs which discriminate the instructed groups 
are descriptive of performance following instruction. Since no difference 
were found on the Demonstration d i mension- ~ the performance aspect of the 
assessment procedure--we lo^oked closely at the Demonstration behaviors 
whLcli appear to separate thf uninstructed and the instructed groups, 
witli respect to the two functions. 

The first function indicated that the uninstructed entering groups 
are described by general Social Interaction beliaviors such as maintaining 
an attitude of openness, or iisitig a broad spectrum of interaction skills. 
When th(^ instructed profile is examined, task-oriented skills as well as 
person-oriented skills are evident: assigns tasks (task), maintains eye 
contact (pc/son), indicates differences (person), or stimulates discus- 
sion through questioning (task). Thus, the effects of instruction as 
implied by tiiese behaviors are task- and person-oriented behavior., wliich 
are employed by instructed students for effective group interaction skill 
In the second significant 1 unction, tlie best comparison groups, con- 
trolling for experience, reseml)led each other and are closer on the multi 
variate space. I'he common factor for these two groups is traditional 
college age. This may indicate a maturity factor wh.ch interacts witli ■ 
social I n t e r a c t i o n } ) e t i f > rm a n c e . 

Th..^ two younger groups are ( haracter i zed by person-oriented behavior 
attends lo non-verbal incssagi s , or (Micourages tliose who exhibit with- 
drawing behavior. Are tliest^ l)f>havi()!s descriptive of younger students? 



7o 



The? older uninslructed groups and the contracted instructed group 
are all clustered cOward the negative dimension with respect to U'le 
Second function. A factor that may explain this cluster is motivation 
since the uninstructed groups were older and perhaps more mature and 
the younger group was contracted. The dominant factor might be age Jf 
the 10 Weekday instructed students are excluded. If we exclude these 
10 students, older Weekend College women are characterized by broad 

r 

interaction skills and an ability to stimulate discussion through ques- 
tioning. This skill was identified as an outcome of instruction according 
to the first function. Since it emerged in the uninstructed^group as 
v^ell, attributing it to the effects of instruction may be premature. 
But the fact that young, instructed students perform this ability as 
well as mature women, given the nature of the ability under study, 
argues for the validity of instruction. 

Since on both functions, older, uninstructed women are described 
as having broad interaction skills, and sir ce broad interaction skills 
signify level 1 of the Social Interaction competence, it is reasonable 
to assume that women entering Weekend College r^ay need -less extensive 
instruction on Social Interaction level 1 behaviors. * It is suggested 
that faculty examine the possibility oc shifting instructional of forts, 
towarci the higher levels of Social Interaction for these groups. 

The faculty member who scored t[ie generic instruments for this 
study useci information about level 1 performance from t[»e external 
assessors' evaluation of student performance at level 1, The fad 
that uninstructed students demonstrated high pf/'r forma nee on level I 
behaviors according to external assessors' judgment, implies that 
external assessors are validating students quite readily. This issue 
of lower standards at leve! 1 for entering students shouJri he t'Xamined 
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as well as why instructed sLiulents did not pe r f o rm ^li ghe r on level 1 
behav Lors . 

When Leadership) characteristics are examined across all five groups, 

♦ 

the analysis yit^lded a list of leadership behaviors which hold constant 
across all groups and could be considered a generic profile of leadership 
ability, which includes person- and task-oriented skills. Thus, ari 
emerging leader in a group could not be effective in focusing only on 
task-oriented behaviors and ignoring person-oriented behaviors, nor 
can she focus only on interpersonal behLaviors and ignore t a s k-or i (Mi t ed 
beliaviors. This supports the Social Interaction Division's rec(Mit 
incorporation of a person- as well as a task-oriented model. A 
balanced approach to both ])ers()n- and task-oriented behaviors will 
more likely produce an eff(u^tiv(^ leader. 

The cuiitracted, instructed group showed a marked reduction in leader- 
sliip behavicirs. It was assumed that performance uruier the pressure of 
validation produced a conflict of i n te res t-- ach levement of group task 
vs. effective individual performance for credent ialing purposes. Under 
such circumstances, all students are trying to exhibit as many behaviors 
ns possible. There may be no occasion to demonstrate other leadcMship 
behaviors which are other-or i ent (m1 in nature^. This argues for an 
assessment technique that elicits more c'o 1 1 abo ra t i on . 

In conclusion, the discriminant analysis assisted in establishing 
construct characteristics of tlie major two groui)s: instructed vs. 
uninst.ructed . It showed clearly that uninstructed students iriteri)ret 
(effective Social Interaction skills differently from instructed students. 
If the instruct(Mi group descr i i)t ion sui)ports the instructional iramewurk, 
th(ui faculty may develop ari instructional program which will take into 
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account the baseline i)ehavi(n\s of entering students and further (ieve.lop 
the desired hehaviors which did not descril'e the instructed groups. 

C r i t e r i a V a i i d i t y 

This section of the analysis dealt witli criteria evaluation. Cari 
we identify erileria which best discriminate instructed and unins true ted 
groups? A combined group comparison where the total instructed group 
response was compared to the totaJ unins true ted groups', response yielded 
three discriminative criteria: 

• Assigns tasks to help implement decisions or attain consensus 

• Maintains appropriate^ eye contact, with everyone in the group 

• Attends to non-ve rha J messages and cues 

A paired group comparison yielded an additional criterion which 
appeare(1 to discriminate betwee^n the two groups jiroviding the most 
accurate comparison for i ns t rue ted/iin i ns t rue teti . 

» 

• Stimulates discussion through effective^ g^J-^^l related 
quest ion i ng 

All urri ns t ructed groups performed consistently higher on: 

• Develops a chx vjno logical strategy for assistir^g the group 
towa rd goa I a ch i.i*vc»nyu] t 

• IdtMitifies tu rengths and weal'uessc^s oi' own pert ormanct^ 

• Points out ways to imi)rove one's own c I f ecti vene, <^ . 

The faculty coder felt that the tOrms un i r»st r rct(Ml students lilled 
out prior to Gi(nip Discussion (Preparation dimension) were uvL comparable 
to the forms completed by instructed ^a.ud(MIti; . '1 he iOiaiier J-timulus 
elicited "a ch lono 1 og i ca 1 strategy lor assisting ^ e grc^up lowards g )a I 
aclvi evemen t." more explic- ly, Tlni.s , it cannot bc^ cone luded t.[i,it 
un i ns t rue t c^d si i<ierits are more <»lleetive in devrloping < h r (uh> I og i ca i 
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ttMnstcr ()\ .ihiiilir;, .ir t ii . i i ! \' < •, c i ■ n r jr-o f iniu < j , i s s <•(>' ;hi 1 ( ■ ,m" e; : n )» t<' I fir 
ICS siiniiLiti.m (\x('rrisr iX'S^ i ;• , t n init t: 1 viliMilv ^-iw hr ; 1 1 1 ? ■ s i. i , . ; i , . 1 . 
Aii t. .itc'(i tM.iwr, tlif /V'l.rt 1 n ;, ! ri ]iui M ! t, i:; .ni .i I. t » Mil] ^ t ^rciL^- i 

romnujn iiii';: :..n rr ,i( ^(r^;^, ,j 1 1 i ,i I intrt.K \ ion , i i • :;ii!rn I. p i . : ■ r ^ i n r r ■ ; ■ .il 

Ifvcl 6. fl' )v,f ";r r- , si[H(^ thr !(''•; Srnup 1 ) i :-> m :-. i u n (Mlris .! lift! ir]\{ 
.^sse^;sMU'[ll p i'm rr • 1 u ri' t i <hii u'ii.il i oitrrrJ r-t\mi 1 .i i- ! v i li ( la;,sr;;, i -Li [i t i n.^y. 
t.lli^; ru'jsurt' whi'rli w.ij; p^ciir r.i t rd { fuin f 1 ;i :-■ ri n hd . m ?, i v i t u ' :; l ^ • .i !ir\., 
ipen - oiKit'd s ! I. lir'i I. i (MI is 'pir:; t. i (m.il) 1 , Dnt's the li'S '-roup [) i s ( ii n mi 
(' ! i 1 I t.hr hrli,) V 1 ( ' i"s vin t. [i Avr ^lt'.^^;l! r('f! Ii\ Uir j^riuTic i ii ^; I, r unit • ri I 
Tlif'se (jtu 1, i (')n s rcf) f^-MWMtt b.-isii issues of t. hr i n^-; I r-iiiiirn I rrl.il MMisliip. 
In I he assrssiiit.MiL iiuxit^ ( > i' Ic Ih'' .i s '.M-; s.iiim n t. st iiiini : . 

A um 1 t. i p 1 f.' r r r ( M> ^ ; 1 . ) M m n ; i v s i s w ;i r ni p ] f > \m ' ( i t_ * ) j s ; i s I * . j < ■ m ! v id 
i (if-Mi I i 1 y I i r i l r r i .i wh m, h i on l r i luit ( ^ in or ni.i v imp t( - vr t. 1m ■ [> rri 1 i < i 1 1 mi 
of Uir <liinfMisi(>n score: eflrclive iM-epM r m I i ( mi , eltrtt. ive !)t'n,< Ml^; I la t i uii , 
eft(H-tive SfM { -Asses sm(Mit. . Tiiis lyi-' ('i mmmIvsi:; will assist. Uie So^iai 
InLeraetion Division to y.cvn in <>n liie " i inpe r t a n l hehav- im; wh i h uinler 
lie efreclive Preparation^ (Ml(M'tive Deinorisirat i eai , an^l elte(tive Se li - 
As seSvSnKTi t , I'ariilty t Ihmi can qu.i 1 i t a t i ve I y evaluate the rema i ii i rp\; 
erileria and f In ir i"e } ,i t, i orisli i p t. < > the- dimension tonslru(t. and a!:o 
identify basf^line pe r To niia ri( e o\ th(> behaviors. it will m ! so aesist 
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Til i Slihly :>r-\TMMl w \ o \- n ii t i o'.ii ; ' s \o]- cnir V/oik ill V .i 1 i ( 1 , i I I ll 

. J s siiU'U t I f ' ( hii 1 <|ii(' ^. . I'lisl . il Itwl us lo v I'v 'fid coiilra.sl sever. jI 

.s I .i I i s I i CM I iiirtluMis ioi Jiialy/in^; lUc rvM^uIts t roni .m /jLlenipt lo '/.ilifl.it 
St. ill juoIIkm iiKMsmc. Srceiitl, I he liii»iiii^s have several i mj) 1 i i. ,» I 1 1 »ii s 
! ( ..• J V ,i 1 i < I a t. 1 < ) ! 1 ' ' t 1 1 M • r M u I a in ) ;e i ; e r , i 1 . Tl i t r t i , v; e ( . i iiie t », ^ heller ii n * 1 e r - 
si ami the natuie of the Stir i .i ! I iiL(M"acM i en <."eitipe I etu e , lea<liii)\ t. • * t. lie 
.'.J 1 1 <la I i on ri t I he eoinr f > I eiue as well as the leeluii^iiH^. 

The tir.*;l t i nd i ie<', i :> thai ,t.'.i'<^iip eeiiipa r i S(Hu; [)C^l.we(M t )a d i I i < 'm i - 
a >;<'(! iiesliauted ami urilni e uii i iis I rue I e<i s. ludcait. s may iu)l he aii eHeeli\'( 
st.!.ilex',y tei va 1 I (la t I u V, assessment t(H"hni(]ues li Ihe reuipetaMire i eiie 
lirvt'leped thrrui>:;li iuloriirjl is well as collr'ge IcsaniiiiK- Sneh r eiiijx't eiua 
m.iy .ils.e luteiact with level et ej^o (ie ve 1 opmeii L . And such a strategy 
may hr {jremature whc^re littU^ is uride rs loud al)ouL the natiii-e o 1 t lie 
ah i lily. lu >^eiuM\t 1 , l-hc> stu<iy indicated similarities in pe r f e iiuaiiee 
hetween inslrueLe^ and uninstiiuted sLudeuLs in a p re- / p o s t t es t design. 
WtHMi we ignor-"d hf^ exUMit te wri i c h students had hcuMi insliau led 
huW(»ver, <nid ronihiued hoth groups in a <l i s cm* i in i ria u I. analysis, some 
c^lear findings tmu^ ged . We t ourid that soiut- ot tho Scjcial 1 nt.e r a e L i can 
abililit^s did inde(Mi separate* Idn^ un i ns t rn c:l (mI sf idenLs- as a group Iroin 
the* insliueted sludcuts. ddie cril.eria lliaL d i s c r i in i na t ' l.he instructed 
students do show evidence of ■ (^ffCc^ts of i r)s t nu" t i on , in lh<it thc^y 
arc more clost^ly related to those aspects of SocMal hreractiou tliat ar-t 
Ifarned as part of a nu,)rc^ spcM:if ic social interaction process. 

Realizing Lhc^ possi' le effects ow pc» r f o rinanc-e of such variables as 
age, lyue of program, and prior college experience, wo tried to control 
for these variables at first. Such an undertaking, given the nature^ ol 
tho compoton(^e we were assess ijig, was in retrosj)ect dooiric^i t(.) failure. 



\)vy>\)\\r '.'Hi ct torts, v/f wct^^ snnply not .itno lo rnfiLroI tor nic luyri.id 
rMn>.-,(' of vnri.ipie:. t, h.il .m-(> likrly to .iflooi ihc rostiit^;, .irid t ho bov', iii- 
nuie, sort. io[i I h i s j)<f]w.T i j iiian\' dosi r'lptiou of ,i ] 1 ih(> oilier 
possibio ni^p(M'l.s f llir safn[^lf\ t lio to.idi lions o[ thf.^ a s s s sine ti I , .jrni 
Uio ass(:'S^""irM) t iwocoss iImI vorr at least, [jarlially uru on I. ro ! 1 ed . WIhti 
thI^. iS ronibiiied o ; h the nature- oi comp<'l(Mic-e under study, vhero 

f lo 1 o rnia in.r -il inlioarts ^^ i t. Ii a :;ol r> 1 porsona! oud <;xo dovr I opruoi) l 
vari.iblt^s, I ry I n>.', I" sej.ai;ato 'at I lie sjuM/if ic el i cuts of instruct ion 
thai show s i gn L !' 1 ca ni dMt'tM~enees between small saiiii:ilt: sizes tlirouj^h 
,^roup eoinparisori is rn^t an ellcHtive st.tate^y. 

^jroup eofn|)a r 1 soiui , wfier{^ persf)[no a r'e diflirult to eategorixi^ and 
matching rs i iiiju a c t i ca 1 because of small samf)le .sizt^s, may not be tlie 
l)esl strategy for evaluating tlu^ effects of instruction. One out(.-(v.ue 
of the analysers, then, is that a strcuig focus on gr^nip comparison^ wliere 
the gr oups are i iidc^ptniden t rather thari longitudinal, niay fiot be an eifec 
tive strategy iinless the groups can be matched. An(.)thcn" go.i 1 , to iuclu(i(* 
a broad range students on age ami exneriencc^ did lead to an effective 
strategy for identifying those Social Interac-tion criteria that validate 
the (.oustruct. (letting at the co; truct is the most im])ortant task. The 
collects ot instriatic)n .Me also s iiiportant , btit in Social Interaction, wc' 
do red yc^t kncjw eai'Migh about tho integration of <jgo and experience witii 
p,t^ r t a rmancc . C^ristraun validity should tak(* [)riority in sucdi c-ases. 
This is probably b(vst accomp 1 i sl^ed through mu ! t i v«ir i a t.e ratfier than 
un i va r i a t e tec. hn i f jues . 

In the main, our analysis sinews th,at the study of assessnuMit techni- 
<|uc\s should not f)e limited to linivari^ate j';ethods . Even if (iiti'erenc- 
do not appear in grouj) conipa r i s«^)ruT , a great deal can be learru'd <ibont 
the construct unrler study, and lb" ef fee's' of i nst met i ui . (',(Miij^a r od to 
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mj 1 var 1 .il <• <i 1 y s » iiiii I I i va r i a t e analyst\s givr us a \uovv hulistM 
virw of tlu' ])t»rrnrinaii(<' rrjdc. sUnly. It tail assist 'is in i dcii I i I y mi.u 
pm;()i' ra t. f'^o 1 i , aiul pallcriis oi coiiercMit jut i o r'ni<irirf> Lh.it 

is:H'r-i;r in tht' d.ila. This also provides us v.ilh a mure liolisl.it picUirf 
o{ ^rnu|) (ha ra r t e r i s I i cs , whtM-'.'as a univariate ari lysis ran lix^k at 
duly one facloi- ,,t a tmu^. This is particularly i.ipniiaul in relation 
tc- ct Mj:-: 1 :-ur t valiTiiV .siin. f wish lo t'X,uni[ie the s jin i 1 ; r' i I i t v. and 

Tilterences in pr r 1 o rm.ni ei^ thai Teserihe the dUlerent gr-onps. From 
tlie re^r'ession analyses, v,'e can examine the kinds oi roust rncts 
.e;:;rssors are irsin)> to evaluate sliuients. Such analyses t;ive us 

^iiinpse nt the e r i t r i a an assessor eni{)hasize^ in descrihir^K 
s I uden t pe r- f o nna n ee . 

in ediMMtiuu, i(N!n" st.ntii(\s .ire sehioin yiossihle ^iven limita- 
tions such as thosc^ (h-serihed in the des- t.^j) seel ion. Matihed saiui)lfs 
or 1 ong i t ud i.na 1 p re- / po s t. le s t designs may be? i inpos s i h i.e . Puttjiig 
all stu(ients into the same analysis allows us to see which groupings 
(uenr in th(> dat.i, even though the r(>sult^: from the discriminant 
auaiysis must he i n t e rp r(^ t erl with caution and results used as 
indicators rather than conclusions. 

hi our prior s^auiy oi the V.ilumg and Commnii i ca t i ons c~oaipeteuc(-^ . 
we a<l{)pted tlu- »dea of studying a wid(> range of students in ordcM" to 
investigate construct validity. In the present stiuiy, this strategy 
provided the most iiselul information for both construct validity and 
i US t met i ona 1 des i gn pu rposes . 

With respect to an overall design, wv do recommend, for fntur(> 
studi(-^,a Longitudinal p!-e-/post test design with a particular class or 
t rom one cot.vpetence level unit to anot.her. Wh i h> a i or.gi tud i na 1 stmiy 
ov»-r two y^'.irs may liot work because of the eontirnial rcMlesign 
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nu- r i 1 11 1 urn <inH assessnu^r\t Lcchri i. <|iirs , we might fOliow one j^^rouj) t roni 
level ! lo level 2, aiiothei' troni level 2 to level 3, another group i roin 
lev(»L 3 to level A, aiul st) on. I're-/posl t est d'.'s i gns eoi.'ld be used 
beater ar a partieiiiar strategy i-^sr stii.iyiiig cii.uiges in studtMit })crfonnanee 
v.'ithin ecuirses, even tliough effects may not n(»cessarily be attributtul 
to specilie asserts of instrurtion in a partieular course because ol tiu? 
lac'k <}{ a eonlrol group. 

The issue re is t.he value of two grouj) comi^a. r i sons . At this level, 
it has been ine, 'ctive. By tlie time we sp(M:ify a 1! the qualifications 
to the results we begin to wonder why we are doing i. iie study. Clearly, 
tiiese comparisons need to control variables. . But. the Intensive st\idy 
of which variables contribute to the variance in performance takes 
considerable time and effort. Such study is best left to external 
validation efforts, rather tliaa to internal efforts, in our opinion. 
Further, faculty need to be able to predict performance from those 
variables, but at the individual student levf^l . An interview by an 
experienced faciilty member can quickly elicit information on prior 
college experience, and other levels of understanding, and the 
ii.struct may then use the information to prescribe instruction. 
Identifying qualifications for explaining results is a good exercise^ 
however. It reinforces one criterion for the assessment process, the 
need for muitii^e samples of behavior. 

r'etbo(' ^ 'Jgy chosen for validation needs to he set within the context 
of whei instrument is in the validation process. At the same time, 

oa; carino* wa't until an instrument is "perfect" before one initiates 
>t unties of validity. We need an ongoing process, and sometimes, given 
t^c necessity of information for the validity of t.he construct ot tl)e 
competence, we will validate an instrument for that purposes. 
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Ch'Mtly, this sttnty stip{)orls llir nrtM.l lor our inpl.ici^, r.x t « mi s i ve • 
I'V.i 1 u<i t. i oil / ri'V i s 1 oil sysUni ( Assc^ssnuMit. CtJinnn I U^v'^^l H (. o <>{ Kesearrli an*] 
l.v.i ' i.i I i on , wli 1 (. 1) ( oMil) 1 no;; j vMriety o* slralo^ios aiul (U'poinh; 

it. 1 1 u 1 .1 r 1 V ( \>' lovoi i>i validity ot ass(ssiji(Mit tocniiiqur. Tins 

sy . fp, ioco^ni/'S that all i n s t rmiu'iU s arc always i [i procoss. At any oiio 
jv.inl .n I 1 iiir , .in i ii s I i uiiio n I may liavo a [la r t. i (ai 1 a r 1 ('Vo < ot vaii>ii(v. 
i- ■■ ox.iFHpit^. )l iii.iv liavo hooi! M-V](ns'0(i hy nno iiiomfx^r- <»f t. tio d op. i r I inr n (. 
,f '■ us(Mi (^no soinoslor so lliere^ is [)r r iO rin.i n c o data availal)lo. 
Al arothor Irvol, il may havo boon rov i cwt^i hy I ho AssossiiKUil lannu i 1 
.u',.Mnsl t ho oriloria lor hrlj'.in^ i ns I ruinont s , arrl it iiKiy liavo ha-i a 
criloii.i <"V.i 1 MaM on . Al slill anot.hor lovoh, intor-tator roliahilily 
of assessor- jndgiiionls may havo f)oon o s I <i h 1 i s h' 1 , and slndont perl or- 
manoo analy/od lOr vnpul inlo triloria tor sUidonl c r rden t i a 1 i n . 

In ^;um, an import. iiu/l hoiio I <jk v cm ! stralt^gy tor hc^^inniiiK work 
on v.i!i(iation ot an i n s t r liiiuai t moasnring a oompoloiHa" ttiat. is not v.a^ 1. 1 
,i(»iinod, is to rolloot 1 n 1 o niia t i on t roiii a wide rango ot stndents wlio 
differ on lovel ot inatnrMty, oxporioncc and instruction and examine 
tlieir per tDrniantn* to identify those criteria wlii^li discriminate the 
eata.'^or i zat ion ol individuals: This will give us a [)etLer idea of tlie 
holistic nature (.>t per form. iiua on tlie compet one a.' . Anottier ixMietit is 
that we will ttien he .11)10 t(j i(ieiitity tlie criteria that are m)%k likely 
to provid(^ faculty w i t ti indicators tliat a^student is inastc-ring tlie al)i]ity 
One pr(jl)lom in designing criteria is choosing those criteria from a graaaf) 
which give the best picture of t lie ability under study. SucTi a discrimi- 
nant analysis can assist with this effort. 

In Cfjiiclusion, the present -tudy ontliiHMl a proc:edure hy which Ihc 
integration of information ab-out a coni[)ca(MH c construct, dif ."rent grou]) 
ch<i I acter- i s t i cs and tluMi- i e 1 .i t i onsh i p to t)eh.-i v i o ra I criteria, and criteri 




evaluation contribute to an information base for instructional develop- 
ment, re-evaluation of competence behavioral criteria and revision of 
instruments which meas .re these behaviors. 
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